From c7bbb09311d57fe17147aa00dc6ec44a341c0cf1 Mon Sep 17 00:00:00 2001 From: Foued Attar Date: Mon, 12 Oct 2026 15:26:47 +0200 Subject: [PATCH] =?UTF-8?q?Ghost=20of=20Y=C5=8Dtei=20and=20Tsushima=20in?= =?UTF-8?q?=20game:=20rendering,=20GPU/CPU=20memory=20coherence,=20tessell?= =?UTF-8?q?ation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * [ShaderCompiler] Pass guest LDS size to SPIR-V compilation * [SpirvTranslator] Synchronize wave64 half masks through LDS exchange * [Fiber] Avoid formatting unused names and disabled traces * [Shader] Build materialization cache ranges without temporary copies * [ShaderCompiler] Intern scalar graphs without string keys or hit allocations * [Media] Vectorize movie YUV conversion without changing pixels * [Shader] Reuse scratch storage for resource read recording * [Tests] Initialize presenter state for GPU binding validation * [Vulkan] Keep temporary descriptor writes off the managed heap * [GPU] Reuse bounded descriptor preparation scratch arrays * [ShaderCompiler] Persist versioned scalar analysis across launches * [GPU] Remove duplicated image layouts from texture bindings * [ShaderCompiler] Bind cached graph payloads to their input keys * [Vulkan] Load content-addressed driver cache shards on demand * [Tests] Keep presenter allocation fixture independent of diagnostics * WIP: Yotei runtime descriptors, compile-cost fixes and Codex memory work before PR #1000 Snapshot before merging upstream PR #1000: runtime image descriptors (global table), format-store and fault-flush compile fixes, runtime buffer strides, stencil plane fixes, Codex's image memory, D16 and residency work. Tests: all pass except the two known failures (DescriptorReloadedInsideLoop_SharesOneSampler, RuntimeFormattedLoad_ReadsItsElementThroughThePageTable). * [Cpu] Claim guest thread stack and TLS regions atomically TryMapGuestThreadRegion looked for a free slot and mapped it in two steps, and Map reuses a region that already exists. Two host threads claiming a nested-callback stack at the same time therefore got the same stack, and Map zeroed the first one's frames. In Ghost of Yotei two render threads ran the AGC command-buffer-full callback on that shared stack at once: one returned through the other's entry stub onto the other thread's host stack, abandoning managed frames. The runtime then failed fast with "attempted to call a UnmanagedCallersOnly method from managed code" or an access violation, one to four minutes after the intro. The slot search and the mapping now run under one lock shared by the stack and TLS claims. * [Gpu] Preserve downloaded buffers and batch resource descriptor reads * [Vulkan] Forget driver cache identities of destroyed shader modules The remaining change of the updated #1006 head (e6659681) that the integration branch did not carry. * [Vk] Measure GPU busy time from per-submission timestamps Per-draw timestamps are written only with SHARPEMU_PROFILE_GPU_DRAWS=1; by default each submission gets two timestamps and the overlapping intervals are merged into a [PERF][GPU_BUSY] line with the busy share and the longest submission. * [HLE] Reserve a host core for the render thread On Windows hosts with 8 to 64 logical processors the render thread gets the last physical core of the fastest efficiency class at the highest priority, and guest threads are kept off that core so spinning job workers cannot take the renderer's SMT sibling. SHARPEMU_RENDER_CORE=0 turns it off. * [Vk] Spin briefly before blocking on a GPU wait A GPU wait polls the timeline for SHARPEMU_GPU_WAIT_SPIN_US microseconds (150 by default) before it sleeps, which saves the wake-up latency when the work completes soon. * [GpuMemory] Answer CPU-dirty range queries without the tracker lock MayHaveCpuDirtyPages reads the per-block summary and the region masks without taking the region lock; a true answer still goes through the locked path. Regions also bump a CPU-dirty epoch whenever a page becomes CPU-dirty. * [Gpu] Skip the device-address sweep while no page became CPU-dirty PrepareBda remembers the CPU-dirty epoch and the mapping it swept last and skips the full sweep of mapped memory until either changes. * [GpuMemory] Scan possibly CPU-dirty blocks 64 at a time ForEachPossiblyCpuDirtyRange combined a region lookup and a summary read for every 4 MiB block of mapped memory on each device-address sweep. A bitset of blocks that have a region and the dirty summary words are now scanned a word at a time, with the same runs as before. * [Gpu] Skip syncing a written buffer range that is already GPU-owned SynchronizeBuffer returns early when a written range lies entirely in the GPU-modified ranges, uses the lock-free CPU-dirty query for reads, and the render profile splits buffer sync time into written, texel and upload parts. * [Gpu] Version the GPU-modified buffer ranges A written range bumps GpuModifiedVersion only when it adds bytes to the GPU-modified set, so readers can cache a clean verdict for a page until the GPU writes something new. * [Gpu] Track the extent of DCC metadata and answer overlap queries A color target records the size of its DCC metadata, and the image cache can tell whether a guest range overlaps registered DCC metadata. * [Diagnostics] Trace the GPU writers of DCC metadata and of a chosen address SHARPEMU_TRACE_DCC_WRITERS=1 writes the shader program and a call stack for GPU buffer writes that overlap DCC metadata; SHARPEMU_TRACE_WRITER_ADDRESS does the same for writes that cover one guest address. * [Gpu] Fill DCC metadata on the GPU and in guest memory at once A fill that targets DCC metadata which no GPU write owns is recorded as a GPU fill and also written to the guest backing directly, so the range never becomes GPU-dirty and later CPU reads need no readback. Ranges the GPU still owns or that overlap an image keep the plain buffer fill. * [Gpu] Turn AGC fill compute shaders into buffer fills BoundedFillDetector recognizes the AGC range-fill and pattern-fill compute programs, including thread-dimension dispatches, and a dispatch of one that targets DCC metadata becomes a DCC fill instead of a compute dispatch. Refused conversions are reported by the DCC writer trace. * [HLE] Resolve backing aliases for readers that hold an access window SharedBackingViews publishes its view snapshot as an alias identity, and a reader can enter an alias access window once and resolve guest pages to their backing alias without the views lock until the snapshot changes. * [Diagnostics] Trace shader resource reads that wait for GPU-written memory SHARPEMU_TRACE_GPU_READS=1 writes the shader, stage, address and table phase of a resource read that has to synchronize GPU-written memory, once per shader and page. * [ShaderCompiler] Keep recent materialization variants per draw key A cache key now holds up to four materializations in most-recently-used order, so alternating descriptor contents reuse their earlier result instead of re-materializing. The cache report counts stale and unreadable entries and table refreshes, and ReadingTable tells a reader when the flattened table is being evaluated. * [VideoOut] Read resource words through cached clean pages and the backing alias TryReadGuestWord and TryReadResidentGuestBytes remember the last 64 pages found free of GPU-dirty bytes, keyed by the GPU-modified version, and read them through the backing alias inside the alias access window that each command stream slice now holds. GPU-dirty checks are byte-precise, and a resident read refuses unreadable ranges before it asks the page tracker. * [Gpu] Read GPU-written guest memory back asynchronously and ahead of time VulkanAsyncReadback keeps eight slots: slot zero serves synchronous reads and the others carry asynchronous downloads. A guest thread that reads GPU-written memory lets the render thread start the copy and waits for it on its own thread, then the render thread applies it if the range is unchanged; a foreign read keeps the source buffer alive until it completes. Windows the render thread had to read synchronously become hot. When a batch writes into a hot window, the window is downloaded right after the batch is submitted, and the next read uses that copy if it is still current. A pending write to a hot window ends the open batch early. SHARPEMU_ASYNC_GUEST_READBACK=0 keeps the synchronous path; [PERF][ASYNC_READBACK] reports the counts. * [ShaderCompiler] Prefetch contiguous resource words in one resident read Resource materialization read every descriptor and flattened-table word through its own guarded TryReadGuestWord call, about 2.6 million per five seconds in Demon's Souls. The raw reads of each plan are grouped once; at run time their addresses are resolved, sorted and read as contiguous runs through the resident guest reader, and the values seed the evaluator cache. A run the GPU may still own, or one out of bounds or through a null base, falls back to the word reads. The cache recorder records the prefetched words, and SHARPEMU_RESOURCE_PREFETCH=0 turns the prefetch off. * [Gpu] Reuse image lookups that found the same backing FindImage remembers, in a 256-slot table, each request that resolved to an image with the same backing, keyed by the request bytes and a generation that every index registration or removal advances. A repeated request touches the image and returns the remembered result without walking the page owner index. SHARPEMU_IMAGE_LOOKUP_MEMO=0 turns it off; [PERF][IMAGE_LOOKUP] reports hits and misses. * [ShaderCompiler] Look up fused programs without the registry lock Every shader header lookup and decode asked the fused-program registry under its lock. The registry now publishes an immutable dictionary that readers take with a volatile read, and a registration copies it under the lock. * [ShaderCompiler] Decode fused programs whose halves are far apart The fused decoder joined the continuation at its address distance from the entry and refused halves more than 4 GiB apart or in the wrong order, which Poppy Playtime now hits for a vertex shader (invalid-fused-layout). Such a continuation is placed right after the entry instead, and each of its instructions keeps its real offset from the program address in AddressOffset; S_GETPC_B64 in the scalar graph, the SPIR-V and the MSL translators uses that offset, and the persisted scalar analysis key includes it. * [Gpu] Collect an image only once it is old enough in frames Since images age by frame, clamping the age to the current frame made every image of the first frame collectable during the first 16 frames, and a submission that ended no frame could delete a texture the frame still used. Without memory pressure nothing is collected until the full age has passed; under pressure the clamp still lets collection relieve memory early. * [Tests] Cover the deferred global barrier of an attachment-only scope A global barrier still ends a scope whose draws store to memory. In a scope whose draws only write attachments it waits for the scope to end, and a later memory-writing draw ends the scope and records it. * [Agc] Make the number of suspend points in flight configurable CommandStreamQueue takes the number of frame boundaries that may wait behind the graphics commands. The default stays one, so the guest CPU can run one frame ahead, and SHARPEMU_SUSPEND_POINTS_IN_FLIGHT=0 makes a suspend point wait until its own boundary is processed, for diagnosis. Tests cover both modes, including a label that the CPU rewrites after the suspend point. * [Gpu] Throttle the device-address fault and 3D slice range logs Little Nightmares wrote 195 thousand "Accessed non-GPU cached memory" lines in one session and Silent Hill 6800 slice range lines. Both now print the first occurrences and then only at power-of-two counts, with the count. * [Docs] Document the new environment variables Adds SHARPEMU_ASYNC_GUEST_READBACK, SHARPEMU_GPU_WAIT_SPIN_US, SHARPEMU_IMAGE_LOOKUP_MEMO, SHARPEMU_RESOURCE_PREFETCH, SHARPEMU_SUSPEND_POINTS_IN_FLIGHT, SHARPEMU_RENDER_CORE, SHARPEMU_TRACE_DCC_WRITERS, SHARPEMU_TRACE_GPU_READS, SHARPEMU_TRACE_WRITER_ADDRESS and SHARPEMU_PROFILE_GPU_DRAWS, and corrects the group counts to the rows the tables hold. * [GpuCommands] Write the 64-bit label of a compute-done release with an interrupt RELEASE_MEM with event 0x2F and a 64-bit data selection was accepted only without an interrupt. It is now handled like the other end-of-pipe events: the label is written and the interrupt is raised after it. * [GpuCommands] Let a wait see its own queue's preceding label store A memory wait that directly follows a 4- or 8-byte store of its queue to the same address, with only waits or inert NOPs and markers between them, compares against the stored value when guest memory no longer holds it. Dead Cells writes a label and waits for it while the CPU already rewrites it for the next frame. The forwarded value lives only within one slice, and any draw, dispatch or transfer in between drops it. * [ShaderCompiler] Key fused programs by the underlying guest memory Each native guest thread wraps the guest memory in its own tracking wrapper, so a fused program registered while one thread created the shader was not found by the render context. The registry now unwraps ICpuMemoryWrapper before it looks up the per-memory table. * [Agc] Implement the legacy sceAgcFuseShaderHalves export The older NID (nApJjpKNBl4) shares the fusing path but keeps the front half's user data pointer, takes the larger value of the resource allocation fields of both halves, takes the user SGPR count and the ES component flag from the front half, and copies the checksum register of the matching stage. * [ShaderCompiler] Decode S_CMOV_B64 as a conditional select S_CMOV_B64 is decoded as S_CSELECT_B64 with the old destination pair as the false source, so it reuses the SCC, register-pair and EXEC handling of the translators and of the scalar value graph. * [Gpu] Merge the stencil replacement references of a face Vulkan has one stencil reference per face for the comparison and every replace operation. Replacements with the test value and with the operation value are now merged into that reference over the written bits, and refused only when they conflict on bits the comparison also reads. A replacement with an operation value of zero over the written bits becomes a zero operation, which ASTRO's PLAYROOM uses to clear one stencil bit. * [Json] Add the sce::Json::Value copy constructor and keep shadow values on copies The copy constructor (fSb2oQTNrgA) shares the assignment path. Assignment and reads now carry the value a constructor or setter recorded in the object heap, so a copy keeps it after the source is destroyed. * [Font] Implement sceFontTextSourceInit Fills the 0x60-byte text source with the text range, the cursor, the parser callback and its object, and the text size, and leaves the memory after it untouched. * [AudioIn] Answer sceAudioInGetSilentState without an input backend No AudioIn port can be opened until an input backend exists, so the export returns an AudioIn error code (0x80260101) instead of being an unresolved import. * [PhtreadSpecificSemanticsTests] Fixes for computed shaders * [Gpu] Return idle buffer memory and bound retired memory under pressure - Small-buffer slab chunks track their live blocks; completely idle chunks go back to Vulkan under memory pressure instead of staying allocated until device teardown. - Buffers removed from lookup still hold GPU memory until their retirement completes. The cache now counts them, collects on registered plus retired bytes, and waits for completion once retired memory passes 256 MiB. - The tiler's pooled scratch buffers are released under pressure, and image collection waits for completion when outstanding scratch is over budget. * fix(shader): preserve wave64 ballot masks on Vulkan * test(shader): cover subgroup lane bits in wave mask predicates * [Pad] Give each new touchpad contact its own touch id The pad assigns a new id (1 to 127, then back to 1) to every finger contact and keeps it while the finger stays down. SharpEmu reported the finger slot instead, so lifting the finger and touching elsewhere read as one contact jumping across the pad. Games that tell strokes apart by the id (Ghost of Yotei's name drawing) never saw a new stroke. * [Gpu] Stop device-address faults from churning the buffer cache Shaders reach guest memory through pointers, and a page they touch outside the cache faults and is registered afterwards. In Ghost of Yotei that path re-created tens of thousands of buffers every few minutes: - Faulted ranges outside the GPU mappings were never touched again, so they aged out while still in use and faulted again. They are now kept and touched each tick, and forgotten once their guest mapping is gone. - A fault inside a guest mapping now brings in its aligned 2 MiB window, clipped to the mapping, instead of one 16 KiB page. - Faults on memory the guest never mapped (bad pointers) no longer create buffers: the shader reads zeros there, as on the console. They were 98% of the faults and about 2.6 GiB of junk buffers; the cache now holds ~0.5 GiB in the same scene. - The per-fault log line is gone; it wrote hundreds of thousands of lines per session. * [Vulkan] Save and load driver pipeline caches past 2 GiB Ghost of Yotei's driver cache grows past 2 GiB. Saving went through a .NET array, so every save failed and new pipelines were lost at the next launch; loading had the same limit. Both now stream the unchanged file format from and to native memory in chunks, so existing caches still load. * [Np] Treat NO_ENTITLEMENT for unowned add-on content as an expected result sceNpEntitlementAccessGetAddcontEntitlementInfo returns NO_ENTITLEMENT when the user does not own the add-on, as on the console. The import log reported it as a warning. * [Vulkan] Log only slow pipeline creations Every pipeline creation was logged, cache hits included, which wrote thousands of 0.0 ms lines per session. Only creations of 500 ms or more are logged now. * [Shader] Treat a phi that merges one value as that value when planning table reads * [Test] Read the formatted-load V# from GPU memory so the load goes through the page table * [Repo] Add license headers to new files and build pipebench with central package versions * [Test] Add the missing copyright line to the large dispatcher tests * [Repo] Remove game-specific RenderDoc capture scripts * [Repo] Remove the pipebench tool * [Kernel] Implement sceKernelAioPollRequest * [ShaderCompiler] Return the entry header address of a registered fused program An overload of TryGetFusedProgramParts also gives the entry header, so a caller can register the same fused program on another memory. * [Vulkan] Keep saving a pipeline cache that outgrows its limit A driver cache over 256 MiB was refused on every save, so it was never written again and every launch compiled the same pipelines. Such a cache is now written as an empty file once, and the next launch starts a fresh one. A save is skipped when the driver reports the size that was loaded or last saved, and a failed or deferred save waits for the next interval instead of running again on every new pipeline. * [Vulkan] Write only the cache shards that gained pipelines A shard remembers the driver size it was loaded or saved with and is not written again while that size holds. Shard saves are logged as one summary line, and a shard load only when it is slow. * [Vulkan] Remove the single-file cache once the shard mode takes over The shard mode never reads the single cache file of the default path, so it is deleted instead of staying on disk. * [Gpu] Record compiled compute programs in a shader prewarm list When the pipeline host gives a list, each compute permutation the game compiles is appended at once to shader-prewarm.bin: the guest code the decoder read (once per program), the fused parts, the compute input, the system registers, the resource specialization and the push data start. Records carry a checksum, so a cut tail is dropped at the next open. TryCompilePrewarm rebuilds the compile request from a record through the same layout and request code as the runtime, so it produces the same SPIR-V. * [Vulkan] Prewarm the recorded compute pipelines after a translator or driver change A first run after a translator change compiled every compute pipeline one at a time while the render thread waited (Demon's Souls: 282 pipelines, 144 s). When the translator assemblies or the driver differ from shader-prewarm.stamp, background threads at below-normal priority translate the listed programs again and create their pipelines, so the driver has them before the game asks. With the same build and driver the launch does no prewarm work. The log reports the prewarm result and how many runtime compute modules it produced. * [Vulkan] Keep the pipeline cache in memory on Windows The GPU driver keeps its own shader cache on disk, and the shader prewarm list rebuilds it after an update, so Windows no longer writes the driver cache file by default. SHARPEMU_VK_PIPELINE_CACHE=file keeps one; Linux keeps the file and macOS keeps its shards. * [Docs] Document the pipeline cache modes and the shader prewarm list Adds SHARPEMU_SHADER_PREWARM, SHARPEMU_VK_PIPELINE_CACHE_SHARDS and SHARPEMU_VK_PIPELINE_CACHE_LEGACY_SHARDS, and describes the new cache file default. * [DeS] Fixes for page faults and stack bounds issues in the emulator and audio fixes * [ShaderPrewarm] Hold the game until the prewarm finishes and resume interrupted runs The prewarm compiled thousands of compute pipelines on background threads while the now faster boot ran beside it, so logos, Bink movies and the first scenes raced the compile and dropped frames. - Hold the guest at its first splash hide, GPU submission or CPU flip until the prewarm is decided and done; the render thread keeps presenting the splash. SHARPEMU_SHADER_PREWARM_WAIT=0 opts out. - Stamp only the shader translators, the compiler type and the driver, so a Libs-only build no longer triggers a full prewarm. - Record finished pipelines in shader-prewarm.progress so an interrupted prewarm resumes with the remaining ones. * [AudioOut2] Block a blocking push only until a queue slot frees A blocking sceAudioOut2ContextPush waited only when the host stream hit its 128 KiB cap, so a title rendering faster than real time kept about 680 ms queued and every sound played that far behind (GTA V: queue=2 context, queued_ms avg=678). The push now waits until the device holds the cushion plus QueueDepth-1 grains, the level GetQueueLevel already reports as full; the host cap stays as the limit for non-blocking bursts. GTA V now holds about 41 ms with no underruns. * [ShaderCompiler] Share the device-address formatted load decoder per component count Every BUFFER_LOAD_FORMAT through a device address inlined the whole page-table walk, format conversion and bounds check. Shaders with many such loads grew to several megabytes of SPIR-V; 128 D16 loads added about 4600 words each. Emit one DontInline function per component count, taking the address, size, offset, descriptor and caller PC. Fault recording, formats, swizzles, EXEC and out-of-range values are unchanged. Offline A/B on three extracted shaders (RTX 5070, driver cache disabled): 18.6 s -> 2.9 s, 12.9 s -> 2.8 s, 15.9 s -> 3.2 s. * [Gpu] Never copy a linear image readback past its guest range The readback storage is sized from the guest range, but the copy follows the linear layout computed for the format, whose padded rows can need more bytes. A 513x513 R8 image declared as 0x40401 bytes was copied with 768-byte rows (0x60201 bytes): the copy wrote past its slot in the download ring, and at the end of the ring into freed memory, which lost the device. Plan the readback only when the copy fits the guest range (and the partial-level readback only when it fits its covered bytes), with one warning naming the image. A linear download that would still exceed its destination now fails explicitly instead of corrupting GPU memory. * [Gpu] Draw merged LS/HS tessellation through a native tessellator The guest hull runs unchanged in a cooperative wave64 compute workgroup and computes its own control points and tessellation factors. A generated control stage feeds those factors to Vulkan's tessellator, and the guest domain shader is translated to a tessellation evaluation stage that reads the guest control-point ring. - VGT_TF_PARAM gives the domain, spacing and winding; VGT_LS_HS_CONFIG gives the patches per group and the control-point counts. - Factors are clamped by the guest's minimum and maximum levels; zero or NaN outer factors still cull the patch. Isoline factors are swapped into Vulkan's order. - The SPI_SHADER_USER_DATA_ADDR_HS pair is carried separately from the LS scalars and moved to s0:s1 before the guest code runs. - Patches are drawn one hull group at a time, because the guest rings are reused by relative patch id. - A configuration the bridge cannot express (power-of-two partitioning, an undefined domain, an empty hull group) skips the draw with one warning instead of stopping the emulator. * [VideoOut] Verify a guest page once for clean resource reads Resource materialization reads a few hundred guest words per draw through the clean readers, and each read re-checked the whole GPU ownership of its bytes: GPU-dirty pages and bytes, then the image cache, which built a new list (twice per word) under its lock. In Ghost of Yotei that was about 550 million reads and 90 GB of allocations per minute in game. - The image cache answers whether an unsynchronized GPU image covers the bytes without building the list; the list is built only when one does. - The clean-page memo, already used by plain reads, now serves clean reads too. A page is verified clean once (no GPU-dirty bytes or pages, no GPU-modified image) and trusted until a buffer or an image is written by the GPU, which advances the buffer cache's or the images' write version. GPU-dirty pages are still checked on every read. Measured in game: one materialization went from 0.33 ms to 0.17 ms. * [Gpu] Keep the guest's face culling on tessellated patches A patch list stands for legacy rectangle lists, which are never culled, so every patch-list draw dropped its culling. Tessellated draws use the patch topology too, and their output rasterizes as ordinary triangles: back faces were drawn. Only a patch list without tessellation stages is a rectangle list now. * [ShaderCompiler] Give the domain shader its patch ID in v8 The ES ABI passes u, v, the patch within its hull group and the patch ID in v5..v8. v8 was never written. Each native draw covers one hull group, so the patch ID is the group's first patch plus the native primitive ID. * [Gpu] Run tessellation hull groups together in their own off-chip slots The hardware gives each hull threadgroup in flight its own off-chip buffer and its own run of the factor ring, so groups run together and one draw consumes them. Every group used slot zero instead, which serialized the draw into one dispatch, one draw and full barriers per group. - sceAgcDriverSetHsOffchipParam (VGT_HS_OFFCHIP_PARAM) is kept: buffer count and granularity. Ghost of Yotei sets 256 buffers of 8K dwords. - Hull workgroup k runs group k: its first patch, its patch count, s2 at slot k and s4 at group k's run of the factor ring. - The domain shader finds its group from the native primitive ID: the off-chip slot in s4, the patch within the group in v7, the patch ID in v8. - One dispatch and one draw cover as many groups as the off-chip buffers and the factor ring hold. Without the guest's configuration, one group runs at a time as before. * [Tests] Restore two upstream tests the branch dropped without a behaviour change DepthState_PlayroomReplacementClearsOnlyTheWrittenBit and FormattedUntypedLoad_SpecializesOnTheDescriptorFormat were removed by earlier branch work although the code still behaves as they expect; both pass unchanged. * [Tests] Check the AudioOut2 port volume and reserved bytes again The layout change dropped the checks of the -1 volume at +4 and the zeroed bytes at +8, which the export still writes. * [Repo] Remove the branch's debugging switches The capture-after-seconds RenderDoc request, the crash-capture directory and first-chance fault traces, the unsupported shader dump, the AGC allocation failure traces and the control-flow trace served earlier investigations only. The configuration switches (direct memory size, image cache budget, forcing the shader dispatcher) stay. * [PlayGo] Do not reject chunk IDs the playgo.pgm numbering only assumes playgo.pgm is read for its chunk count only; the IDs 0..count-1 are derived, yet they were marked authoritative and any other ID returned BAD_CHUNK_ID. The list is still reported, but like other metadata that is not understood, it no longer rejects an ID outside it. * [Vulkan] Bound the bindless sampler binding by the device's sampler limits The persistent heap always declared 4096 update-after-bind samplers and left them out of the pool-wide descriptor budget it split between sampled and storage images. On a device with fewer samplers per stage, or a tight update-after-bind total (MoltenVK, some Intel drivers), creating the heap could fail. The sampler binding now takes the smallest of the device's per-stage and per-set sampler limits (at most 4096), and its share comes out of the total before the images split the rest. * [Gpu] Copy linear render targets with the guest's own pitch A linear render target's builder lays it out with the guest's pitch, which need not be aligned like a texture's: a 513x513 R8 target is 513 bytes per row. Transfers recomputed the texture's aligned linear layout (768-byte rows), so its uploads read shifted rows and its readbacks did not fit the guest range and were skipped. A single-level, non-volume linear image whose description holds a consistent slice layout now transfers with that layout. Textures, which carry no layout of their own, keep the format's linear layout, and the readback guard still refuses a copy that does not fit. * [Gpu] Free images and retry once when an image allocation runs out of memory The image budget comes from the device's reported memory, so an allocation can still fail when the driver or another process holds more than expected. That was fatal. Running out of device or host memory while allocating an image's backing is now recoverable: the cache retires every image collection allows, waits for the retired ones to be destroyed, returns unused pool memory and tries once more. A second failure stays fatal. Tests make an allocation fail through an internal hook on GpuDeviceInfo. * [Vulkan] Size descriptor pools for the sets that need them The branch had raised every descriptor pool to 65536 storage buffers, 262144 sampled images and 8192 storage images so the largest shader's set would fit, and every pool paid for that on every device. Pools are back to upstream's ordinary size (8192/8192/1024/1024). When a set does not fit one, the next pool is sized for four sets of its kind, and a retired pool is reused only if it can hold the set. MaxImages stays the planner's bound on distinct image descriptors per shader. * [Vulkan] Blend with a second source from the MRT1 export Blend factors 15-18 (SRC1 color and alpha) were mapped to Vulkan's Src1 factors without enabling dualSrcBlend or giving the fragment shader a second output, so the blend read an undefined value and validation reported every such pipeline. When the device supports dualSrcBlend, it is enabled, and a draw whose target 0 blends with an SRC1 factor (and binds no other target, as on the hardware) gets a second fragment output fed by the MRT1 export, declared at location 0 with index 1. The translator accepts that one extra output and keys the compiled program by it. * [Tests] Skip device tests without a Vulkan device, as upstream does The branch had turned the device gates into assertions, so every device test failed on runners without a GPU. SHARPEMU_TEST_REQUIRE_DEVICE=1 still makes a missing device a failure. * [Gpu] Wait for pending image readbacks before uploading over them A tiled image retired under pressure publishes its contents only when its readback tick completes. Found again before then, it was rebuilt from the guest bytes it supersedes. Uploads now wait for the readbacks into their range. * [Shader] Unpack compressed 16-bit pixel exports by their export format Compressed exports were always unpacked as half floats. UNORM16, SNORM16, UINT16 and SINT16 exports (G-buffer normals among them) now follow the SPI_SHADER_COL_FORMAT of their target, which also keys the compiled program. * [Agc] Emit direct draws for every draw modifier sceAgcDcbDrawIndex and sceAgcDcbDrawIndexAuto returned without writing the draw for any modifier other than 0x40000000, so whole passes vanished without a trace. * [Agc] Emit addressed occlusion counter events sceAgcDcbEventWrite refused every event with an address, so the 0x38/0x39 counter events never reached the command stream. Their results are now published through the ordered buffer fill path instead of CPU memory only. * [Gpu] Execute target-less draws whose pixel shader writes memory A draw without color or depth targets was dropped before compilation even when its pixel shader stores to buffers or images. Yotei fills its visibility buffer this way; the dropped pass left most of the scene black. * [Shader] Keep EXEC guards after V_CMPX with a legacy scalar destination GFX10 V_CMPX in VOP3 or SDWA form changes EXEC and ignores its scalar destination. The EXEC-full analysis took it as a write of that destination, kept EXEC full and elided the guards of the following vector writes. Ported from voidborn-astra/sharperemu 8499b905. * [Vulkan] Use unsigned operations for scalar pair bit instructions Ported from voidborn-astra/sharperemu c0cff157. * [Gpu] Invalidate image copies for raw buffer writes Ported from voidborn-astra/sharperemu caa4149d. * [Gpu] Retain GPU image contents during collection A GPU-written image whose readback a dirty buffer overlap blocks was cleared and deleted, losing its contents. It is kept instead; same-page CPU writes are resolved by hash before deciding. Ported from voidborn-astra/sharperemu 3b5eb2f3. * [Shader] Read the selected lane for V_READLANE in graphics stages Graphics stages took the reading lane's own value instead of the selected lane. A graphics wave is one group of up to 32 host lanes; the selected lane of that group is now read with a subgroup shuffle. * [Tests] Cover Yotei's wave64 march and waterfall loop shapes A step counter compared into EXEC with lanes leaving through VCC, and a waterfall over an SDWA compare into an SGPR pair. Both already end every lane in both halves of the wave. * [Vulkan] Make depth bounds dynamic pipeline state The depth bounds were part of the graphics pipeline key. Ghost of Yotei moves them per draw, so every value created a new pipeline: about 80 per frame and thousands kept for the session. They are now set with vkCmdSetDepthBounds per draw; the key bytes they used stay reserved. * [Shader] Read absent graphics lanes as the reading lane A host subgroup of a graphics stage can be partly empty, where a PS5 wave still has dead lanes. DPP, V_READLANE and V_PERMLANE reads of an absent lane returned undefined values: Ghost of Yotei's wave minimum then never reached its exit and its pixel shaders ran until the step guard. An absent lane now reads the calling lane's value, which keeps min/max/and/or reductions exact. * [Memory] Skip the ownership transition for pages already CPU-dirty A CPU write to pages that are all CPU-dirty re-ran ChangeState, which scans the whole 4 MiB region and republishes the same summary, for every managed guest store. CPU and GPU dirty states are exclusive, so nothing can change there. * [GPU] Keep the previous frame's images through mid-frame collection Collection runs before every draw while image memory is under pressure. It evicted every image not used yet in the current frame, which the rest of the frame samples again, so Ghost of Yotei recreated and uploaded about 7000 images per frame. Below the critical level the mid-frame pass now leaves them alone; frame-end collection and allocation-time reclaim are unchanged. * [GPU] Copy the newer GPU contents between same-layout image aliases One surface read through two formats of the same layout is two cache images over the same guest bytes. Before one is used as a texture, storage image or target, it copies the newest GPU contents of its same-layout aliases on the GPU, ordered by write sequence, as unified memory shows them. Depth keeps its reinterpretation path. * fix(kernel): provide read-only entropy device descriptors (cherry picked from commit 2ad0eca819ba8456b42177d4e682915d99fb7639) * fix(net): track epoll descriptor creation and destruction (cherry picked from commit e1c5658ed5f9579756111c89943b136fb995182b) * fix(pthread): implement relative timed mutex acquisition (cherry picked from commit c75f28f489a7fa3f78d6baae533e49c33dc10cb7) * fix(net): implement level-triggered socket epoll waits (cherry picked from commit 2f27f0078888007861f71afc5615e5c9a7246532) * fix(kernel): release timed-out event flag waiters (cherry picked from commit 4daa8a34e4c0534d3a0ebcc79f7a15597aaed330) * fix(system-service): report empty event polls (cherry picked from commit 65fb9aae7a7ee994ce8ab5b70f8fb00df2f7c742) * fix(videoout): preserve decoded frame presentation order (cherry picked from commit 66526f0eedc4da4253a6650b5563c257ff5c278d) * fix(ajm): decode ATRAC9 jobs from split buffers (cherry picked from commit e8c9925d86eb886d8734913a0ba7c10c125cc209) * fix(videodec): finish queued frames before decoder deletion (cherry picked from commit eae19e7c170998af3f4cf865759536fb33886e28) * fix(net): confirm socket errors before reporting epoll error readiness (cherry picked from commit 95128f67d446b4f659692584d241bdc85a5a9af7) * fix(net): format guest Ethernet addresses (cherry picked from commit 08cca779b5c000b46322ce16eb75b9739f312175) * fix(net): keep one sceNetEtherNtostr export Both branches added sceNetEtherNtostr. Keep the version that reports an unreadable source address as a fault, the errno a guest read fault gives. * fix(net): distinguish pending accepts from socket hangups (cherry picked from commit 28ad0c0a9d989ad8f368d471354ede295ce9acf7) * fix(net): avoid querying payload bytes on listening sockets (cherry picked from commit 2aac142b2fb58b4b398bb1f2dfdf96cd94a388f1) * fix(audio): avoid pacing empty nonblocking context pushes A nonblocking push returns at once. The grain it claims still counts in the queue level when no host backend is attached; only a blocking push waits for its slot. (cherry picked from commit 9b2db6d814f89b44cf46da11c3f8ced474ace063, adapted to keep the queued grain accounted) * chore(net): remove an investigation trace call (network part of 8d0bd2fc; its PlayGo trace only exists in the skipped d342dd1c) * fix(videodec2): preserve the picture-info size prefix Without a picture only the ready flag at offset 8 is cleared; the caller's size prefix and the other fields are preserved. A decoded picture still fills the whole output info, pitch included. (cherry picked from commit 050e9c824ad4a1ac2d9a40a6d6dd9ba1f63fd4fb, adapted to keep the complete output info for decoded pictures) * fix(videodec): preserve adjacent picture-info fields Width and height are 32-bit fields with the pitch between them. (cherry picked from commit 88a79715; the complete output info this branch writes for a decoded picture already used 32-bit fields, so only the dimensions helper and its test are added) * fix(shader): report descriptor accesses that fall back to the null descriptor A control-dependent image or sampler descriptor with no plan-time source reads the null descriptor, so its texels read as zero. The plan records each such access and the host warns once per shader with its PC, so the loss is visible instead of silent. * fix(shader): read control-dependent descriptors at run time before the null fallback A sampled float access whose image or sampler descriptor has no plan-time source read the null descriptor even when the runtime descriptor table could serve it, because the null fallback ran before the runtime path was tried. Such an access now reads its descriptors at run time, as the hardware reads them from memory; the null fallback stays for accesses the table cannot serve yet. * fix(gpu): keep invalid runtime image descriptors on the null texture A shader reading its descriptors at run time reports every image it does not find. One with a null base or a format without a host equivalent stopped the emulator when the host tried to create it. It now stays out of the table, so the shader keeps reading the null texture, as the hardware does for such a descriptor. * fix(gpu): translate equivalent stencil replacement states (cherry picked from commit a6e7922dbede9548fe5ef8795d29387de98b1bf8) * fix(gpu): accept GFX10 image resource-level bit (cherry picked from commit 1cbafc0a52e44592e34c5ae708d98573f6d2e9d6) * fix(gpu): preserve addressed texture view alignment (cherry picked from commit 936d873f70922337b7e9557cf4e579f4ed73d471) * test: keep only the tests that 1cbafc0a and 936d873f add Resolving their test conflicts kept test blocks from later #1057 commits that depend on code not merged yet; only each commit's own test remains. * fix(gpu): read disjoint descriptor bytes through backing aliases GPU ownership is tracked in bytes but page protection covers whole pages: CPU-owned bytes on a page that also holds GPU-owned bytes are read through the backing alias, without bringing the neighbouring GPU bytes back. (cherry picked from commit 7c8f0b97; this branch's clean word reader, which first synchronises GPU-owned words, is kept, and the test keeps the resident-range assertions) * fix(gpu): read indirect tessellation draw arguments on the CPU Non-indexed indirect draws now leave their arguments to the GPU (#1036), but the merged tessellation path counts patches from the vertex count on the CPU: an indirect tessellation draw arrived with the placeholder count of one and drew no patch. * fix(gpu): write a replaced stencil plane back before dropping its proxy A depth buffer whose stencil plane has another size at the same address drops the other buffer's stencil proxy. When the GPU had written that plane, deleting the proxy stopped the emulator (Ghost of Tsushima). The plane now reaches guest memory first, as when a depth buffer changes planes, and stencil write-backs register as publishing downloads so an upload of the range waits for them. * feat(shader): read image-load descriptors at run time An ImageLoad whose descriptor has no plan-time source looks its words up in the runtime descriptor table instead of reading the null descriptor. The key carries the numeric class the descriptor format selects, so the shader fetches through the uint, sint or float heap array the host view matches. - Only active lanes record misses; an inactive lane may hold anything. - A missed descriptor over unbacked memory stays out of the table, and an entry whose memory the title freed leaves it when its image is retired. - An entry keeps the texture it resolved to until the cache retires its image, so a draw no longer looks every entry up again; each draw still acquires its views. - A plan that reads descriptors at run time on a host without the bindless heap is a rejected plan, not a fatal layout error. * fix(shader): resolve descriptors from empty scalar buffers Descriptor words read from an empty scalar buffer are zero whatever the offset, so the access reads the null descriptor; materialization rechecks that the buffer is still empty. Ghost of Tsushima's pixel shaders read image descriptors this way. (cherry picked from commit 5d03a71aacdead6adfbc9f46b5eeb777168e759a; on this branch an access the runtime descriptor table can read keeps that path, which also serves a buffer that is not empty, so the proof applies to the other accesses only. Its tests use a depth-compare sample accordingly, the indirect-image case stays a runtime read, and the planning-dump fixture adds one to its descriptor as f73663e8 does upstream) * fix(shader): preserve scalar sources in immediate comparisons (cherry picked from commit 9f4e325df34d9c075d9625da9994ecccfa9d1ac3) * fix(shader): count unsigned leading bits in scalar instructions (cherry picked from commit 09e9e1de7a8de12d92de39d51eed744ff4416fa6) * fix(shader): bind signed integer storage images (cherry picked from commit 1c134311d66ea6690a4549dff933017e1e478c5a) * fix(shader): view only 32-bit signed storage writes as unsigned A write-only signed storage image went through an unsigned view for every signed format while no signed storage binding existed. With signed bindings (1c134311), only the 32-bit format keeps that view, where the bits are the same; a 16-bit signed image stores through its own signed view again. * feat(shader): read adjusted-sample descriptors at run time The IMAGE_SAMPLE_*_A forms carry their adjustment in the sampler's reserved dword-3 bits. Their descriptors without a plan-time source now go through the runtime table like a plain sample, with those bits masked out of the sampler key so each adjustment does not register a sampler of its own. A Ghost of Tsushima pixel shader reads the image of such a sample from a material record that the plan cannot prove. * fix(shader): qualify single-mode sample interpolants (cherry picked from commit 3026bb288124f001f828f550755a2e4117c67ec0) * fix(gpu): preserve sample masks and decode signed locations (cherry picked from commit 3ab11b99d98482694037365c4e3865839544d891) * fix(gpu): distinguish attachment sample counts in pipeline keys (cherry picked from commit 51dd915d42f2ec877df970b66e0f7d2fc57b1300) * fix(gpu): preserve EQAA depth and overrasterization controls (cherry picked from commit 1087b9fba3914d3afda21041897c223dda3f324d) * fix(gpu): retain scan mode control for sample iteration (cherry picked from commit c8614eae54bc66d18ee60ac594280b9659b1fdd4) * test(gpu): verify captured mixed-sample register decoding (cherry picked from commit 943fc08e17f11dbe0995849813900a666781d42c) * feat(shader): support explicit pixel interpolation samples (cherry picked from commit fa66d55425a6108760f5dc1f7d6d5e9f62c2b8d9) * fix(shader): honor eligible early depth testing in Vulkan (cherry picked from commit 9b34fa404eaca555170e9c5fb18622a907b70179) * fix(vulkan): preserve pixel sample-mask exports (cherry picked from commit 5fbe6a3b38c1724c8c66fcbb6eb37aacdde4a981) * fix(vulkan): preserve pixel depth exports (cherry picked from commit bad8e123e566cf0a1a835fd5fa8736625ffa9581) * fix(gpu): separate pixel iteration rate from interpolation inputs (cherry picked from commit 9e0bdd4cea8105243347772328ec69a0a77e7b11) * fix(vulkan): interpolate fixed custom samples with explicit offsets (cherry picked from commit 8e7796636bed4fad5d5c65ce04a35b970fc369be) * fix(vulkan): gate fragment execution with post-depth sample coverage (cherry picked from commit 4c981e8edfae91b0a9ecdc336a2813ec841941f6) * fix(gpu): apply sample exclusion to early fragment execution (cherry picked from commit 0f5b5a71c54983db73d5d461088f3285cf13641a) * fix(vulkan): preserve per-sample custom interpolation coordinates (cherry picked from commit bfb5d7618b7ec994f2197c8f50d8b717f3c6bca6) * fix(gpu): preserve late depth updates for shader coverage changes (cherry picked from commit 4f8bcb6d142a7352a58d02328d798b390f244ee5) * fix(gpu): key shader programs by custom sample positions (cherry picked from commit dd36fcea9b83fd90ac1aea5af770cfdebbe5f987) * fix(vulkan): preserve two-sample depth with single-sample color (cherry picked from commit 042e5c4d19c5b104f0c20b0a38d87cb349d9241d) * fix(gpu): reject unrepresented two-sample coverage masks (cherry picked from commit b8fe958ac343b049295392a61c99419115020e63) * fix(gpu): preserve forced shader depth ordering (cherry picked from commit 58a4b282c89b2f35d04fe46a3f702c990f5164f0) * fix(gpu): use late depth for unforced shader depth exports (cherry picked from commit 29fe184bf1d241af6e8bfa521020d38d2eaba589) * fix(shader): reject unsupported multisample sampling modules (cherry picked from commit e77451e273d12e9c11355004654d592235e6aa63) * fix(gpu): keep runtime descriptors that describe no view out of the table A T# read at run time holds whatever the lane computed; the hardware reads one that describes no valid view as the null texture. Ghost of Tsushima samples through such a descriptor (image type 0, base level past the last level) before choosing another one. The runtime path now asks for the texture without stopping on it: the T# checks of the texture builder go through one rejection, fatal on the planned path as before and a failure for the runtime one, which keeps that descriptor out of the table. * perf(loader): sample repeated unresolved import warnings (cherry picked from commit 842c3eea5228f017f26941e2f5ad9f69e678c002) * perf(shader): read buffer-table descriptors at run time when the access allows it A descriptor read from a table in a buffer, indexed by a lane value, was planned as a dense table: materialization takes every record the buffer holds and each draw finds, refreshes and binds all of them. A Ghost of Tsushima material table made that about 2000 images per draw, nearly all of the render thread's time. An access the runtime descriptor table can read now takes that path and binds only the records the shader reads; a storage, compare or cube access keeps the dense table. * fix(shader): read sample 0 for samples of a multisample image Vulkan cannot sample a multisample image, and e77451e2 stopped such a module from compiling, which stopped the emulator at a Ghost of Tsushima gameplay pixel shader. A sample instruction on a 2D multisample image now fetches sample 0 of the texel its normalized coordinate falls in, as shadPS4 lowers it, and the image is bound without a sampler. A gather or an arrayed multisample sample is still refused. * fix(gpu): free images and retry once when a buffer allocation runs out of memory Images already retried after freeing what can go; a buffer allocation that found every memory type full stopped the emulator (Ghost of Tsushima gameplay). The buffer allocation now reports the failure, the buffer cache asks the image cache to free its images and wait for retired ones, and tries once more before stopping. * fix(vulkan): initialize format buffer scratch for runtime resources * fix(shader): invalidate stale prewarm resource plans * chore(sync): carry upstream Unity compatibility fixes Apply upstream/main's post-revert Unity compatibility changes while preserving the Yotei-specific AudioOut2 state layout. The upstream rendering reversion is intentionally not imported because this branch still depends on those rendering changes. * fix(shader): bind incompatible-only indirect image tables as typed nulls An indirect image table whose candidates all have a dimension the MIMG instruction cannot address (Demon's Souls 0x9D1D8F5B7F200DE7: every candidate 2D, instruction MSAA) failed with "no typed candidate". Keep the instruction's declared image type and bind every candidate as a typed null instead. * debug(gpu): dump the whole command buffer on an unknown packet Temporary diagnostic for a tester's Yōtei crash ("packet opcode is unknown", header 0x00000000). The old dump showed only 24 dwords around the bad packet, too few to re-parse the stream and find which earlier packet is misaligned. Dump from the start of the buffer instead. * perf(gpu): copy CPU-write-hot pages once per memory visibility point Every draw with a read-only device-address range re-copied the range's CPU-write-hot pages: they stay writable, so they read as dirty forever. In Ghost of Yotei that was ~47% of the render thread (thousands of draws per frame re-copying the same pages, each copy ending the render pass behind a full barrier). The guest only guarantees CPU writes are visible at memory visibility points (a submission slice, a satisfied wait), the rule the PrepareBda sweep already follows. Each buffer now remembers the range whose hot pages it copied in the current visibility generation and skips them there; pages that fault dirty are still copied on every draw. A command-processor write invalidates only the buffers holding the written bytes instead of starting a new generation. SHARPEMU_BDA_RESYNC_HOT_EVERY_DRAW=1 restores the previous behaviour. [PERF][BDA_SYNC] reports visibility points, processor writes and how many syncs skipped hot pages. * perf(vulkan): stop retaining per-draw texture bindings until GPU completion Ghost of Yotei binds ~300 images per draw, and each one allocated a TextureResource of ~1.3 KB that stayed referenced until its submission retired. At under one frame per second every binding outlived gen0, so nearly every GC was a gen1 collection (299 gen1 vs 35 gen0 in 15 s). - Drop TextureResource.Resolution: it was written in one place and never read, and duplicated the ImageRequest (about half the object). - Keep only the host movie planes, which own a staging buffer recycled at completion; the other bindings die with the draw. Measured in game: gen1 collections fell to 9 per 15 s. * perf(images): widen the image lookup memo to 4096 slots The direct-mapped memo had 256 slots, while a Yotei draw binds ~300 images and a frame thousands, so slots evicted each other and only 42% of lookups hit. With 4096 slots (about 2.5 MB) 90% hit. * fix(agc): allocate command buffer space atomically Guest threads run HLE exports concurrently. Two exports appending to one command buffer both read its cursor before either stored it back, so the second packet overwrote the first. A tester's Yotei DCB showed a PopMarker NOP written over the first two dwords of a SET_SH_REG 0x228 packet; the stream then hit its trailing zero dword ("packet opcode is unknown, header 0x00000000") about 20 s into the game. The cursor update is now atomic per buffer through striped locks; the lock is released around the guest's buffer-full callback. A one-time "[AGC][WARN] Two threads append to one command buffer" line reports the race when the same buffer is contended. * fix(gpu): stop executing queued GPU work once guest memory is torn down Closing the window raced the runtime teardown: the runtime detached the guest GPU memory registry while the presenter's shutdown drain was still executing queued draws, so the first device-address draw died with "A device-address program needs the guest GPU memory registry" and the end-of-run reports were lost. The drain now drops the remaining work once the registry is gone. Also report the audio backend a port actually opened instead of a hardcoded "coreaudio" in the AudioOut output trace. * fix(runtime): release guest GPU memory only after the presenter's caches Runtime teardown disposed the guest GPU memory while the presenter was still shutting down on its own thread: its image and buffer caches still held page watches, so PageGuard.Dispose failed fast with "Cannot release the page guard while page state is active". Wait (bounded) until the presenter detaches its stores, and keep the memory mapped if it does not. * perf(buffers): stop read-aheads of windows the GPU keeps rewriting Eager readbacks copied a 512 KiB window after every batch that wrote a hot window. In Yōtei's loading phase and cinematic the GPU rewrites those windows before the CPU reads them: of ~2,900 read-aheads per 10 s, 3 were used and ~2,800 were discarded as stale. Count consecutive wasted read-aheads per window, skip a window after two, and give every window a new chance every 256 batches. * perf(buffers): read only the requested pages from mapped device memory A guest read of GPU-written memory served from a unified (ReBAR) buffer copied the whole 512 KiB readback window out of device-local memory, which the CPU reads uncached across the bus. In Yōtei's loading phase that memmove was 30% of the render thread. Mapped reads now copy a 16 KiB window around the request; copy-command readbacks keep the large window. Loading phase, same 10 s window: QueueRelay 2447 -> 947 ms, frames 205 -> 252. * diag(gpu): report how tessellated draws split into batches Adds [PERF][TESSELLATION] to the render profile: draws, instances, patches, batches per draw and the off-chip buffer count. Each batch prepares and binds every stage again, so batches per draw is the multiplier on tessellation cost. * perf(gpu): run all instances of a tessellated draw in shared batches DrawTessellation looped over instances on the CPU: every instance took its own hull dispatch, a full binding preparation of the hull, domain and pixel stages, and its own draw. Yōtei's cinematic issues up to ~1,900 instances per tessellated draw, ~15,000 iterations per 10 s, and tessellation was half of the render thread. Number patches across all instances instead and batch that numbering by the factor-ring and off-chip capacity, as the hardware distributes the patches of an instanced draw. A new tessellation data word carries the patches per instance; each hull thread derives its instance ID (v5), its patch ID (v0) and its index-buffer vertex, and the domain stage its patch ID (v8), from the shared number. Zero patches per instance keeps the old values exactly. Yōtei: one batch per tessellated draw, cinematic 0.6-0.7 -> 1.1-1.3 FPS. * perf(memory): let CPU-write-hot pages cool down once per frame A page the CPU rewrites right after three uploads turns hot: it stays writable and dirty and every sync copies it whole. Nothing ever cooled a page down, so a ring the game fills a little each frame ended up hot in full. In Yōtei's cinematic two 28-32 MiB buffers were copied whole at every device-address sync, ~60 GB per 10 s. Once per presented frame, hot pages now lose their heat: the next sync copies them once more and protects them again, and only pages the CPU keeps rewriting within a frame turn hot again. SHARPEMU_KEEP_CPU_WRITE_HEAT=1 restores the old behaviour. Yōtei cinematic: uploads ~60 GB -> ~0.6 GB per 10 s, 1.1 -> 1.5-1.6 FPS. * diag(images): trace lost GPU image contents and name guest passes SHARPEMU_TRACE_IMAGE_DROPS=1 reports every time an image loses GPU-written contents that never reached guest memory: the code site, the image, the range of a buffer write that caused it and how much of the image it covers, the image's dirty flags, and the guest render pass decoded from its push markers (one stack per queue). Evictions report whether they published the contents first. A renderdoc.capture file next to the executable now requests a RenderDoc capture at the next guest flip, for unattended runs. * fix(images): keep an image being resolved out of collection ResolveOverlap can allocate a replacement (GrowImage, a depth overlap copy), and an allocation under memory pressure collects images that were not used this tick. The image being resolved is the copy source of that replacement: when it was collected, the copy read a freed slot and the process failed fast with "The slot is not allocated" (seen twice in Yōtei with a 4 GB image cache budget). Touch it before resolving. * fix(images): publish GPU-only image contents before an overlap drops them Memory keeps its bytes when another image or a buffer write reuses part of it. When an overlapping image request or a partial buffer write dropped an image whose GPU contents had never reached guest memory, the contents were simply lost; a later image over those bytes started from stale guest memory. Yōtei's cloud tables alias a 2048x2048 RG16F and a 512x512 RGBA16F at one address, and several post-process targets are partly overwritten by compute buffer writes (7-82 % coverage). Before such a drop, copy the image into the buffer over its range on the GPU (the existing image-to-buffer path) and let the buffer own the bytes, as after any GPU write. Full overwrites still drop directly. SHARPEMU_IMAGE_DROP_WITHOUT_PUBLISH=1 restores the old behaviour. * perf(gpu): poll the timeline at most every 0.5 ms in per-draw housekeeping The completed-submission collection before every draw read the timeline semaphore twice per draw (vkGetSemaphoreCounterValue), about 12 % of the command thread in Ghost of Yotei. The per-draw path now polls at most once per interval and uses the last value otherwise; real waits still read the timeline at once. SHARPEMU_TIMELINE_POLL_US overrides the interval. * perf(gpu): look up known samplers without taking the store lock Every sampler of every draw took the store lock, and the async compute queue contended for it (3.7 % of the command thread in Monitor slow paths). Hits now read a concurrent dictionary; only a miss locks to create. * perf(buffers): read one tracker page per mapped readback Mapped readbacks copy uncached device-local memory at about 300 MB/s, and most want a descriptor word or a packet dword. The window drops from 16 KiB to one 4 KiB tracker page. SHARPEMU_MAPPED_READBACK_KB overrides it. * perf(images): reuse per-thread level scratch when resolving color targets Every draw resolved its color targets with two fresh 16-entry arrays that are only copied into the image description; they were about 10 % of the managed allocation volume in Ghost of Yotei. One cleared pair per thread replaces them. * fix(agc): leave the command buffer alone in Push/PopMarker Ghost of Yotei 1.512 calls the marker exports from a second guest thread on a DCB that the main thread fills with inline-written packets. Appending a marker NOP read a stale cursor and overwrote the game's packets: a tester's dump shows a PopMarker NOP over the first two dwords of a SET_SH_REG, whose stray value dword then failed as an unknown packet (header 0x00000000). Markers are tool-only NOPs on PS5, so by default they now return the current cursor without writing. SHARPEMU_AGC_MARKERS=1 restores the packets for the pass names that traces decode. * diag(shader): say why resource materialization failed Two testers of Ghost of Yotei 1.512 stop on 'could not be materialized ... reason=Other' for one pixel shader, and Other named no step. Each failure site now records what failed and the values it saw (the source and dword that could not be evaluated, the guest address that could not be read, the bad count, stride or descriptor words); the fatal message appends it as detail=. Return values and the fatal/reject policy are unchanged. * fix(agc): preserve marker reservations for the DCB writer Restore packet reservation and capacity callbacks for markers. Suppress markers from another known guest writer under the allocation gate so they cannot overwrite unpublished inline packets. Scope writer identities to canonical memory across native thread wrappers. * fix(gpu): preserve partial images before capturing buffer bindings * fix(audio): report the channel count in AudioOut2 port state * Revert "debug(gpu): dump the whole command buffer on an unknown packet" The full dump identified the tester's crash: a PopMarker from a second guest thread overwrote a SET_SH_REG (fixed by the marker reservation change). Back to the short dump around the bad packet. * fix(shader): probe record-aligned material offsets past the indirect image cap Ghost of Yotei 1.512 stops on 'indirect image 0x494: 72559 unbounded selector probes exceed 65536' for a 2502-record material table with a 116-byte stride. Without a proven selector set every multiple of gcd(stride, 2^32) is probed, because selector * stride + offset wraps modulo 2^32; reaching an offset that is not a record start needs a selector of at least 2^32 / stride. Tables under the cap keep the exhaustive enumeration; past it, the record-aligned offsets are probed and a warning names the image once. * fix(gpu): track guest pages written through device addresses Shaders that store to memory through a runtime V# or a flat/global pointer reach guest memory through the device-address page table, not through a bound buffer. The buffer cache only learned about GPU writes from bound writable bindings, so these stores were invisible to it: the page stayed "clean", the host never downloaded the GPU copy, and the next CPU upload or eviction silently overwrote the GPU result with stale guest bytes (or the CPU kept reading stale bytes). Ghost of Yotei's GPU-driven passes (producer computes filling per-object data and V# tables) write almost exclusively this way. The translator now reports the page of every device-address store and atomic in a second bitmap that follows the fault bitmap in the fault buffer. Non-pixel invocations merge bits of one bitmap word and publish it when the word changes or the invocation ends; pixel invocations, which a discard can end early, publish at once. The fault processor compacts that bitmap with the same pass as the fault bitmap and, once the scan completes, GuestBufferCache.NoteDeviceAddressWrites marks the pages GPU-written exactly as a written binding would: GPU-dirty in the page tracker, recorded in the GPU-modified ranges, the owning buffer's write tick bumped, and overlapping image memory copies invalidated. A page the CPU dirtied before the write is reported cannot be owned by both sides; it stays CPU-owned and the conflict is logged once per power of two. On Yotei these conflicts remain (dozens per run, mostly the producer input pages at 0x50CB...), so mixed CPU/GPU pages can still lose bytes; resolving them needs a byte-level merge and is left for a follow-up. Diagnostics added alongside: - SHARPEMU_TRACE_COHERENCE=1 checks, every fourth presented frame with the GPU idle, that pages the tracker calls clean hold the same bytes in guest memory and in the mapped buffer, and reports each divergence (SHARPEMU_TRACE_COHERENCE_MB sets the slice checked per pass). With SHARPEMU_TRACE_GPU_MEMORY_ADDRESS=coherence the first diverging page is traced. On Yotei the detector went from hundreds of divergences to zero with this fix. - Guest buffers are named "guest 0x
+0x" when Vulkan debug labels are enabled, so RenderDoc captures show which guest range a binding is. * fix(gpu): merge shader and CPU writes that share a guest page On the console the CPU and the GPU write one memory, so a 4 KiB page can hold bytes from both. The page tracker gives each page a single owner. When a shader stored through a device address into a page the CPU had dirtied before the store was reported, the page stayed CPU-owned and the next upload copied the whole guest page, stale where the shader wrote, over the GPU copy. The guest never saw those shader bytes either. Ghost of Yotei hits this dozens of times per run (producer input pages at 0x50CB..., per-draw rings at 0x2002...). SharedPageShadows keeps, per such page, the bytes both copies last agreed on and does a three-way merge: - Uploads of a shadowed page copy only the byte runs the CPU changed since the shadow; the shader's bytes in the buffer stay. - When a device-address write is reported on a CPU-dirty shadowed page (once every upload staged for it has executed), the bytes the GPU changed since the shadow are written into guest memory where the CPU left them alone. A byte both changed keeps the CPU value, which the next upload carries to the GPU. - Downloads refresh the shadow, since both copies agree afterwards. A shader-written page gets its shadow when the CPU first touches it: that access downloads the page, so guest memory matches the GPU copy and no extra read of uncached mapped memory is needed. Pages only the GPU uses cost one dictionary entry. A page the CPU dirtied before its first shader write was reported is shadowed from the mapped GPU copy; shader bytes an upload already overwrote there are lost once, and the log says so. At most 16384 pages (64 MiB) are shadowed. On Yotei: 0 unresolved conflicts (8 one-time losses), 0 coherence divergences, no throughput loss in an A/B run (about 5.6k vs 6.2k draws/s over the same cinematic window). The in-game darkening is unchanged, so these lost writes were not its cause. * fix(gpu): expose the partial last word of byte-addressed buffers Storage buffers are declared to shaders as uint[], and Vulkan's runtime array length counts only whole words. A buffer whose size is not a multiple of four therefore hid its last one to three bytes: byte, short and formatted loads there read zero, and stores were dropped. Ghost of Yotei's occluder producer reads a one-byte-per-instance state buffer of 95 bytes. The last three instances were the ones to produce, so the shader saw them as inactive, wrote no matrices, and the depth prepass drew the object with the previous frame's stale matrices: exploding vertices that cover the scene with black bands and holes. Byte-addressed and formatted bindings now get a view rounded up to the next word. The number of padding bytes travels in the two spare bits of the runtime stride entry (strides use 14 bits), and the shader keeps the guest byte bound for those bindings: byte assembly may read the padded word, but an element is loaded or stored only when all its bytes are inside the guest range, and whole-word accesses keep the old bound. The padded view must stay inside the cache buffer, which spans whole pages, so this never reaches past its VkBuffer. The device test reads and stores the tail of 41- to 102-byte buffers, aligned and unaligned, formatted and byte loads, with and without the zero-out-of-bounds read path, and checks that bytes past the guest range stay hidden and untouched; 13 of its 17 cases fail without the padding. * fix(gpu): only wait for shared page uploads that copy bytes A shadowed page's merge waits until every upload staged for it has executed, since until then the GPU copy lacks the staged CPU bytes. The staged tick was advanced for every upload that visited the page, also when filtering removed all of its bytes because the CPU changed none. Such an upload copies nothing, yet merges waited for its tick, and a page re-staged every frame could keep shader writes out of guest memory indefinitely. The tick now advances only for runs that survive filtering and for a whole-page copy into a new owner. The tests cover an eliminated upload followed by a merge and a real pending upload that still blocks it. * fix(gpu): keep shared page shadows when a buffer grows When a request outgrows a cached buffer, the cache creates a larger one and copies the old buffer's GPU contents into it. Shadows of shared pages still named the old buffer as owner, so the next upload into the replacement treated the page as unknown and copied the whole guest page: stale where shaders had written and not yet been reported, it overwrote those bytes in the GPU copy. The shadows (and the pages waiting for their first CPU access) now follow the copy to the replacement, and a merge waits for the copy's tick, since the replacement holds the old bytes only once it executes. The device test grows a buffer between an unreported shader store and the next upload and checks that both shader writes and the CPU write survive. * diag(gpu): watch guest ranges per frame and force full barriers Two opt-in diagnostics used to trace Ghost of Yotei's stale occluder matrices back to the producer that never wrote them: - SHARPEMU_WATCH_BLOCKS=addr:len[,addr:len...] logs, after every presented frame, each watched range whose bytes changed in guest memory or in the GPU copy (both contents, dirty state, submitted and completed ticks), and every cache event touching a watched range: device-address write reports, uploads, writable bindings, CPU write faults, host DMA writes, fills and copies. - SHARPEMU_DIAG_BARRIER_EVERY_CALL=1 records a full memory barrier before every draw and dispatch, to rule missing guest barriers in or out. It is very slow. Both are off by default and cost one branch when off. * fix(gpu): read and write whole words that start inside a buffer range The previous fix padded only byte-addressed and formatted bindings and kept the floor word count as the bound for whole-word accesses. The hardware range-checks a buffer access by its first byte (stride 0 or swizzle off: offset >= num_records is out of range), so a dword load or store that starts inside a range ending mid-word is performed whole. A dword-addressed buffer of 95 bytes therefore returned zero for its last word, bytes 92..94 included, where the console returns the data; whether the read got zero depended on the driver's robust access granularity, not on the guest descriptor. Every runtime-stride view is now padded to whole words, and a word is in range when it is inside the padded view, which is exactly the hardware bound. Byte and formatted elements keep the exact byte limit carried in the runtime stride entry. The new device test loads dwords from 93- to 96-byte buffers with both read paths; three of its five cases fail without the change. * fix(shader): track both words of binary64 vector results (cherry picked from commit 2e339ac4143061e76bfefb4e065b04eba4647f77) * fix(shader): lower half-word logical shifts on Metal (cherry picked from commit fa9776ec2b71a30c6a833562d8e342f5718a612f) * feat(gpu): pack narrow color channels without changing encoded bytes (cherry picked from commit e98fe0b4e68cd5e070124c97d583911561f5fd00) * feat(gpu): transfer narrow sRGB images through expanded backing (cherry picked from commit 9a5f4e08aaf101ebd68ca18d43efac17ec94112c) * fix(gpu): retain explicit pitch for linear color target transfers (cherry picked from commit 88695316e2119b18cb095ae7ad00e1c17ea1f6fb) * fix(shader): decode explicit-LOD comparison gathers (cherry picked from commit a629cc3949cba63b52cfbdc0b41199a3fc1cba79) * fix(gpu): decode narrow sRGB color target outputs (cherry picked from commit 66aa64a01fc4ad984d53fa248b742481d3b54e61) * fix(agc): accept the shader register Ghost of Yotei 2.002 writes at 0x192 Ghost of Yotei Complete Edition 2.002 applies a shader register table at startup that writes 1 to SH offset 0x192 (0x2D92), just past the LS block. gfx10 documents no register there and no shader state depends on it, but the table writer had no entry for it and stopped the emulator right after the splash screen. Ignore it like the other pace and control registers. * diag(agc): log the values written to the undocumented shader register 0x192 The register is still not applied, but the first value and every change are logged, so a rendering difference in Ghost of Yotei 2.002 can be traced back to what the title wrote there. * perf(shader): look runtime descriptors up through one shared function Reading buffer-table descriptors at run time (90e35ba8) inlined the hash and the eight probes of the runtime descriptor table at every image and sampler access. A Ghost of Yotei pixel shader with 56 such accesses grew from 419 KB to 2,051 KB of SPIR-V and took about 6.5 s to compile instead of 0.5 s. The image and sampler lookups are now one DontInline function each, called with the key and the lane's EXEC state, as the device buffer format decoder already is; the same shader is 507 KB and still binds only the records it reads. * fix(shader): read format bytes in a buffer's last partial dword on the device A format load through a device-side V# assembled each component byte from its aligned dword, and that dword read zero unless all four of its bytes were below num_records. The hardware range-checks the element, so the last one to three elements of a byte buffer whose size is not a multiple of four read 0 instead of memory. Ghost of Yotei's culling updater reads each object's 8_SINT visibility state, skips the store when the new state equals the one it read, and counts the element. Its 19-byte state buffers read 0 for elements 16..18, so the store was skipped while the element was counted: counts[] exceeded the instance slots the producer wrote, and the extra instances drew as black blocks. The dword is now read when any of its bytes is inside the buffer; the existing per-component check still zeroes a component past num_records. * fix(loader): prove jump table bounds and tables over the control-flow graph The red-zone patcher refused every short faultable site of a function whose jump tables it could not resolve. The resolver only accepted a lea-only window between a 32-bit compare and the table load, so Ghost of Tsushima's render thread kept unprotected accesses in a leaf function that holds a pointer in the red zone, and a host exception dispatch clobbered it. Resolve each table by walking the function's control-flow graph backwards from the load: every path must pass the fallthrough of ja after an unsigned compare of the selector, reaching the load through register moves and zero-extensions only, and must define the table register with lea of the same RIP-relative table. The graph includes the resolved table edges, and resolution repeats until a round reproduces its own targets. Tsushima's indirect-branch refusals drop from 55 to 8. * fix(agc): accept the user-config register Ghost of Yotei 2.002 writes at 0x191 The 2.002 build writes 0 to user-config offset 0x191 from a register table, right after shader register 0x192. No gfx10-12 register lives there. Accept it and log value changes, like 0x192. * fix(images): keep a mapped mip chain whole when a texture starts past mip 0 A sampled texture whose descriptor starts past mip 0 was always made an image of only the levels from there, for streamed textures whose larger mips are not mapped. A pass that samples a lower mip of an image the GPU rendered (Demon's Souls' fog and bloom chains) then got a separate image of the same memory, filled from stale guest bytes and traded with the render target every frame: white blocks across the fog and 0.2 fps. The image cache now keeps the resident-levels form only when the chain is not fully mapped; otherwise the texture is the whole chain viewed from its base level, the same image as every other view of that memory. * fix(np): report the package's real trophies from sceNpTrophy2GetTrophyInfoArray b5b3d691 treated rdx as the first trophy id and always returned the requested limit as the count. The call is (context, handle, offset, limit, details[], data[], *count): offset indexes into the package and the count is how many trophies exist from there, capped by limit. Hades asks for 50 and then walked past its own on-stack arrays, crashing in a job thread before the first frame. Read sce_sys/trophy2/trophy00.ucp (unencrypted container with tropconf.json and tropmeta_*.json) and fill SceNpTrophy2Details (1312 bytes) and SceNpTrophy2Data (32 bytes) for the trophies that really exist, all locked. Layouts and the package format follow KytyPS5's NpTrophy2 implementation. Co-authored-by: Felipe <54341095+Pipe530@users.noreply.github.com> Co-authored-by: nmzik * chore(audio): mark the YoteiPC-derived ATRAC9 code as GPL-2.0-only The multichannel block ordering in Atrac9DecodeState follows YoteiPC's at9_api.c, which its author publishes under GPL-2.0 (no "or later" grant). The file now says GPL-2.0-only with the source and author named, and the license text is added for REUSE. LibAtrac9 itself stays MIT. --------- Co-authored-by: ParantezTech <12572227+par274@users.noreply.github.com> Co-authored-by: kuba Co-authored-by: MathiasYG <06mathias6@gmail.com> Co-authored-by: voidborn_astra <41128975+voidborn-astra@users.noreply.github.com> Co-authored-by: Felipe <54341095+Pipe530@users.noreply.github.com> Co-authored-by: nmzik --- LICENSES/GPL-2.0-only.txt | 117 + docs/rendering-resolution.md | 112 + src/SharpEmu.CLI/Program.cs | 36 +- src/SharpEmu.CLI/RosettaGcBudget.cs | 100 + .../Native/DirectExecutionBackend.FastPath.cs | 23 +- .../Native/DirectExecutionBackend.Imports.cs | 52 +- .../Cpu/Native/DirectExecutionBackend.cs | 157 +- .../Cpu/Native/PosixHostStubs.cs | 43 +- .../Loader/GuestRedZonePatcher.cs | 559 +++- .../Memory/PhysicalVirtualMemory.cs | 5 +- src/SharpEmu.Core/Runtime/SharpEmuRuntime.cs | 9 + src/SharpEmu.GUI/GuiSettings.cs | 27 +- src/SharpEmu.GUI/Languages/ar.json | 4 + src/SharpEmu.GUI/Languages/az.json | 4 + src/SharpEmu.GUI/Languages/br.json | 4 + src/SharpEmu.GUI/Languages/de.json | 4 + src/SharpEmu.GUI/Languages/dk.json | 4 + src/SharpEmu.GUI/Languages/ee.json | 4 + src/SharpEmu.GUI/Languages/en.json | 4 + src/SharpEmu.GUI/Languages/es.json | 4 + src/SharpEmu.GUI/Languages/fr.json | 4 + src/SharpEmu.GUI/Languages/hu.json | 4 + src/SharpEmu.GUI/Languages/it.json | 4 + src/SharpEmu.GUI/Languages/ja.json | 4 + src/SharpEmu.GUI/Languages/ko.json | 4 + src/SharpEmu.GUI/Languages/nl.json | 4 + src/SharpEmu.GUI/Languages/pl.json | 4 + src/SharpEmu.GUI/Languages/pt.json | 4 + src/SharpEmu.GUI/Languages/ru.json | 4 + src/SharpEmu.GUI/Languages/tr.json | 4 + src/SharpEmu.GUI/MainWindow.GameOptions.cs | 8 + src/SharpEmu.GUI/MainWindow.axaml | 28 + src/SharpEmu.GUI/MainWindow.axaml.cs | 17 +- src/SharpEmu.GUI/PerGameSettings.cs | 21 + src/SharpEmu.HLE/GpuMemory/GuestGpuMemory.cs | 26 +- .../GpuMemory/GuestGpuMemoryHook.cs | 13 +- .../GpuMemory/GuestPageTracker.cs | 161 +- src/SharpEmu.HLE/GpuMemory/TrackedRegion.cs | 49 +- src/SharpEmu.HLE/GuestMemoryLayout.cs | 13 +- src/SharpEmu.HLE/Host/Posix/PosixHostStubs.cs | 43 +- .../Host/Posix/PosixViewRegions.cs | 13 +- src/SharpEmu.HLE/HostMemory.cs | 109 +- src/SharpEmu.LibAtrac9/Atrac9Decoder.cs | 36 + src/SharpEmu.LibAtrac9/Unpack.cs | 9 +- src/SharpEmu.Libs/Acm/AcmExports.cs | 70 +- .../Agc/AgcExports.CommandPackets.cs | 7 +- .../Agc/AgcExports.ControlFlow.cs | 10 +- .../Agc/AgcExports.DrawPackets.cs | 26 +- .../Agc/AgcExports.DriverResources.cs | 10 +- src/SharpEmu.Libs/Agc/AgcExports.Events.cs | 20 +- .../Agc/AgcExports.ShaderState.cs | 6 +- src/SharpEmu.Libs/Agc/AgcExports.cs | 133 +- src/SharpEmu.Libs/Audio/AjmExports.cs | 136 +- src/SharpEmu.Libs/Audio/Atrac9DecodeState.cs | 490 ++- src/SharpEmu.Libs/Audio/AudioOut2Exports.cs | 68 +- src/SharpEmu.Libs/Audio/AudioOutExports.cs | 10 +- src/SharpEmu.Libs/Codec/Videodec2Decoder.cs | 29 +- src/SharpEmu.Libs/Codec/Videodec2Exports.cs | 5 + .../Gpu/Buffers/BdaFaultProcessor.cs | 75 +- src/SharpEmu.Libs/Gpu/Buffers/GpuBuffer.cs | 27 + .../Gpu/Buffers/GpuBufferAddressBook.cs | 83 + .../Gpu/Buffers/GpuBufferUsage.cs | 2 + .../Gpu/Buffers/GpuDeviceInfo.cs | 130 +- .../Gpu/Buffers/GpuMemorySlabs.cs | 64 +- .../Gpu/Buffers/GuestBufferCache.Watch.cs | 116 + .../Gpu/Buffers/GuestBufferCache.cs | 1275 ++++++- .../Gpu/Buffers/GuestBufferRegistry.cs | 10 + .../Gpu/Buffers/GuestBufferUploader.cs | 12 +- .../Gpu/Buffers/IDeviceMemoryAllocator.cs | 12 + .../Gpu/Buffers/IGuestImageCache.cs | 8 + .../Gpu/Buffers/OptimalImageMemoryPool.cs | 297 ++ .../Gpu/Buffers/SharedPageShadows.cs | 301 ++ .../Gpu/GpuCommands/CommandStreamQueue.cs | 37 +- .../GpuCommandInterpreter.Draws.cs | 10 + .../GpuCommandInterpreter.Events.cs | 24 +- .../GpuCommandInterpreter.Labels.cs | 12 +- .../GpuCommandInterpreter.Waits.cs | 3 + .../Gpu/GpuCommands/GpuCommandInterpreter.cs | 2 + .../Gpu/GpuCommands/ICommandStreamHost.cs | 19 +- .../GpuCommands/Registers/ContextRegisters.cs | 29 +- .../GpuCommands/Registers/RegisterOffsets.cs | 6 + .../Registers/RegisterWriters.Context.cs | 15 +- .../Registers/RegisterWriters.Shader.cs | 21 +- .../Registers/RegisterWriters.UserConfig.cs | 11 + .../Registers/ShaderProgramRegisters.cs | 1 + src/SharpEmu.Libs/Gpu/IGuestGpuBackend.cs | 3 + .../Gpu/Images/CachedImage.Scaling.cs | 135 + .../Gpu/Images/CachedImage.Transfers.cs | 170 +- src/SharpEmu.Libs/Gpu/Images/CachedImage.cs | 160 +- src/SharpEmu.Libs/Gpu/Images/GpuTiler.cs | 106 +- .../Gpu/Images/GuestImageCache.Collection.cs | 204 +- .../Gpu/Images/GuestImageCache.Lookup.cs | 2 +- .../Gpu/Images/GuestImageCache.Metadata.cs | 96 +- .../Gpu/Images/GuestImageCache.Overlap.cs | 109 +- .../Gpu/Images/GuestImageCache.Tracking.cs | 124 +- .../Gpu/Images/GuestImageCache.Transfers.cs | 407 ++- .../Gpu/Images/GuestImageCache.cs | 255 +- .../Gpu/Images/GuestPixelFormats.cs | 5 +- .../Gpu/Images/ImageBackingPool.cs | 26 +- .../Gpu/Images/ImageDescription.cs | 5 + .../Gpu/Images/ImageDropTrace.cs | 100 + src/SharpEmu.Libs/Gpu/Images/ImageRequest.cs | 3 + .../ImageRequestBuilders.ColorTarget.cs | 36 +- .../ImageRequestBuilders.DepthTarget.cs | 15 +- .../Images/ImageRequestBuilders.Texture.cs | 158 +- .../Gpu/Images/RenderScalePolicy.cs | 146 + src/SharpEmu.Libs/Gpu/Images/SamplerStore.cs | 16 +- .../Gpu/Images/SurfaceMetadata.cs | 31 + .../Gpu/Images/TextureTransferLayout.cs | 46 +- .../Gpu/Images/ViewFormatRules.cs | 15 + .../Metal/MetalCommandStreamHost.Rendering.cs | 19 + .../Gpu/Pipelines/BindlessImageHeap.cs | 414 +++ .../Gpu/Pipelines/BoundedFillDetector.cs | 143 + .../Gpu/Pipelines/ConstantFillDetector.cs | 25 +- .../Gpu/Pipelines/DescriptorHeap.cs | 65 +- .../Gpu/Pipelines/DescriptorWriter.cs | 8 +- .../Gpu/Pipelines/IShaderPipelineHost.cs | 34 + .../Gpu/Pipelines/PipelineCacheSignature.cs | 69 +- .../Gpu/Pipelines/PipelineKeys.cs | 26 +- .../Gpu/Pipelines/PixelStageInputResolver.cs | 36 +- .../Gpu/Pipelines/ShaderPermutationDump.cs | 8 + .../ShaderPipelineCache.Tessellation.cs | 76 + .../Gpu/Pipelines/ShaderPipelineCache.cs | 138 +- .../Gpu/Pipelines/ShaderPrewarmList.cs | 4 +- .../Gpu/Pipelines/ShaderProgramCache.cs | 169 +- .../Gpu/Pipelines/StageStaticKey.cs | 16 +- .../Gpu/Pipelines/VertexInputResolver.cs | 19 +- .../Gpu/Rendering/ColorTargetResolver.cs | 7 + .../Gpu/Rendering/DepthTargetResolver.cs | 33 +- .../Gpu/Rendering/DrawTargets.cs | 22 +- .../Gpu/Rendering/IRenderHost.cs | 33 +- .../Gpu/Rendering/IShaderPipelineProvider.cs | 4 + .../Gpu/Rendering/RenderExecutor.Compute.cs | 65 +- .../Rendering/RenderExecutor.DynamicState.cs | 56 +- .../Gpu/Rendering/RenderExecutor.Resources.cs | 63 +- .../Gpu/Rendering/RenderExecutor.Targets.cs | 165 +- .../Rendering/RenderExecutor.Tessellation.cs | 164 + .../Gpu/Rendering/RenderExecutor.cs | 116 +- .../Gpu/Rendering/ShaderStageResources.cs | 47 +- .../Gpu/Rendering/TessellationProfile.cs | 44 + .../Gpu/Scheduling/SubmissionScheduler.cs | 61 +- .../Gpu/Scheduling/TickOwnedResources.cs | 45 + src/SharpEmu.Libs/Gpu/TessellationOffchip.cs | 36 + .../Gpu/Vulkan/NativeHalfConversionProbe.cs | 392 +++ .../Gpu/Vulkan/VulkanAsyncReadback.cs | 18 +- .../Gpu/Vulkan/VulkanGuestGpuBackend.cs | 5 + .../Gpu/Vulkan/VulkanTickDevice.cs | 56 +- .../Gpu/Vulkan/ZeroOutOfBoundsReadProbe.cs | 242 ++ .../Kernel/KernelEventFlagCompatExports.cs | 14 +- .../Kernel/KernelFileExtendedExports.cs | 8 +- ...ernelMemoryCompatExports.BackedMappings.cs | 15 +- .../Kernel/KernelMemoryCompatExports.cs | 193 +- .../Kernel/KernelPthreadCompatExports.cs | 90 +- .../KernelSyncOnAddressCompatExports.cs | 3 +- src/SharpEmu.Libs/Media/HostMovieBridge.cs | 14 + src/SharpEmu.Libs/Network/HttpExports.cs | 70 +- src/SharpEmu.Libs/Network/NetExports.cs | 600 ++-- src/SharpEmu.Libs/Np/NpTrophy2Exports.cs | 213 +- src/SharpEmu.Libs/Pad/TouchContactIds.cs | 36 + src/SharpEmu.Libs/PlayGo/PlayGoExports.cs | 135 +- .../SystemService/SystemServiceExports.cs | 16 + .../VideoOut/HostVideoOptions.cs | 24 + .../VideoOut/RenderDocCapture.cs | 10 + .../VideoOut/RenderPhaseProfile.cs | 5 + src/SharpEmu.Libs/VideoOut/SdlHostWindow.cs | 7 + .../VideoOut/VideoOutExports.FlipRequests.cs | 21 +- src/SharpEmu.Libs/VideoOut/VideoOutExports.cs | 7 +- .../VideoOut/VulkanFloat16Support.cs | 42 + .../VulkanVideoPresenter.CachedImages.cs | 62 +- .../VulkanVideoPresenter.CommandStream.cs | 60 +- .../VulkanVideoPresenter.Descriptors.cs | 349 +- .../VulkanVideoPresenter.Device.Setup.cs | 396 ++- .../VulkanVideoPresenter.Device.Teardown.cs | 9 +- .../VulkanVideoPresenter.GuestBuffers.cs | 4 +- .../VulkanVideoPresenter.GuestDraws.cs | 14 +- .../VulkanVideoPresenter.GuestSubmissions.cs | 4 +- .../VulkanVideoPresenter.HostMovie.cs | 1 + .../VulkanVideoPresenter.Pipelines.cs | 298 +- .../VulkanVideoPresenter.RenderLoop.cs | 3 + .../VulkanVideoPresenter.Rendering.cs | 111 +- ...VulkanVideoPresenter.RuntimeDescriptors.cs | 358 ++ .../VulkanVideoPresenter.Scheduler.cs | 11 +- .../VulkanVideoPresenter.ShaderPrewarm.cs | 15 +- .../VideoOut/VulkanVideoPresenter.cs | 37 + .../Gen5MslTranslator.Alu.cs | 30 + .../Gen5MslTranslator.Pixel.cs | 13 +- .../Gen5MslTranslator.Resources.cs | 5 + .../Gen5MslTranslator.cs | 2 + .../Gen5SpirvShader.cs | 1 + .../Gen5SpirvTranslator.Alu.cs | 276 +- .../Gen5SpirvTranslator.CooperativeWaves.cs | 113 + .../Gen5SpirvTranslator.Interpolation.cs | 124 +- .../Gen5SpirvTranslator.RenderScale.cs | 96 + .../Gen5SpirvTranslator.Resources.cs | 674 +++- ...n5SpirvTranslator.StructuredControlFlow.cs | 1113 +++++++ .../Gen5SpirvTranslator.Tessellation.cs | 154 + .../Gen5SpirvTranslator.cs | 2930 +++++++++++------ .../Gen5TessellationBridge.cs | 102 + .../SpirvFixedShaders.cs | 131 + .../SpirvModuleBuilder.cs | 87 +- .../TilerShaders.cs | 41 + src/SharpEmu.ShaderCompiler/Gen5ShaderIr.cs | 31 +- .../Gen5ShaderTranslator.cs | 56 +- .../Gen5TessellationData.cs | 24 + .../Gen5TessellationInfo.cs | 69 + .../Gen5TessellationLowering.cs | 59 + .../Ir/Gen5ExecFullAnalysis.cs | 3 +- .../Ir/Gen5ExecRunAnalysis.cs | 64 + .../Resources/BindingLayout.cs | 113 +- .../Resources/BufferCandidateTablePlanner.cs | 26 +- .../Resources/GuestImageFormat.cs | 2 +- .../Resources/MemoryAccessInfo.cs | 10 +- .../Resources/ResourceMaterializationCache.cs | 61 +- .../Resources/ResourceMaterializer.cs | 477 ++- .../Resources/ResourceRuntimeInputs.cs | 9 +- .../Resources/ResourceSpecialization.cs | 11 +- .../Resources/ResourceTableReadPlanner.cs | 32 + .../Resources/ResourceTracker.DenseImages.cs | 282 +- .../Resources/ResourceTracker.cs | 235 +- .../Resources/RuntimeDescriptorTable.cs | 163 + .../Resources/RuntimeValueEvaluator.cs | 49 +- .../Resources/ScalarGraphDiskCache.cs | 35 +- .../Resources/ScalarValue.cs | 38 + .../Resources/ScalarValueGraph.Builder.cs | 91 +- .../Resources/ShaderResourceInfo.cs | 62 +- .../Resources/ShaderResourcePlan.cs | 8 + .../ShaderCompileRequest.cs | 51 + .../Agc/AgcCommandBufferChainTests.cs | 22 +- .../Agc/AgcDirectDrawModifierTests.cs | 90 + .../Agc/AgcEventWriteTests.cs | 77 + .../Agc/AgcFusedShaderTests.cs | 8 +- .../SharpEmu.Libs.Tests/Agc/AgcMarkerTests.cs | 302 ++ .../Agc/Gen5MoveRelativeSpirvTests.cs | 53 +- .../Agc/Gen5ShaderAtomicDecodeTests.cs | 39 + .../Agc/Gen5TypedBufferLoadSpirvTests.cs | 23 +- .../Audio/AcmExportsTests.cs | 28 + .../Audio/AjmExportsTests.cs | 186 ++ .../AudioOut2PortGetStateExportsTests.cs | 75 +- .../Audio/AudioOut2QueueLevelExportsTests.cs | 100 + .../AvPlayer/AvPlayerAbiTests.cs | 44 + .../Cli/RosettaGcBudgetTests.cs | 26 + .../Cpu/GuestFastPathStubTests.cs | 24 + .../Cpu/GuestThreadRegionAllocationTests.cs | 72 + .../Cpu/ImportLoopGuardBoundaryTests.cs | 22 + .../Cpu/ImportShutdownTests.cs | 115 + .../Cpu/PosixSleepStubTests.cs | 27 + .../GUI/PerGameSettingsTests.cs | 53 + .../Buffers/DeviceAddressFaultSpanTests.cs | 43 + .../Gpu/Buffers/GpuBufferTests.cs | 37 + .../Buffers/GuestBufferAsyncReadbackTests.cs | 218 +- .../Gpu/Buffers/GuestBufferCacheTests.cs | 293 +- .../Gpu/Buffers/GuestBufferRegistryTests.cs | 25 + .../Buffers/OptimalImageMemoryPoolTests.cs | 59 + .../Gpu/Buffers/SharedPageShadowsTests.cs | 207 ++ .../GpuCommands/CommandStreamQueueTests.cs | 16 + .../GpuCommands/CommandStreamTestSupport.cs | 7 + .../GpuCommands/CommandStreamWorkerTests.cs | 24 +- .../GpuCommandInterpreterDrawTests.cs | 18 + .../GpuCommandInterpreterLabelTests.cs | 6 +- .../GpuCommandInterpreterLoopTests.cs | 24 + .../Registers/RegisterBanksTests.cs | 11 + .../Registers/RegisterWriteTableTests.cs | 145 + .../Gpu/Images/CachedImageTests.cs | 27 +- .../Gpu/Images/GpuTilerTests.cs | 25 + .../Gpu/Images/GuestImageCacheTests.Depth.cs | 150 + .../Images/GuestImageCacheTests.Metadata.cs | 8 +- .../Images/GuestImageCacheTests.Ownership.cs | 106 +- .../Images/GuestImageCacheTests.Readback.cs | 684 ++++ .../Gpu/Images/GuestPixelFormatsTests.cs | 5 +- .../Gpu/Images/ImageCacheTestSupport.cs | 2 + .../Gpu/Images/ImageRequestBuildersTests.cs | 94 +- .../Gpu/Images/MemoryBudgetTests.cs | 25 + .../Gpu/Images/RenderScalePolicyTests.cs | 154 + .../Gpu/Images/TextureTransferLayoutTests.cs | 18 + .../Gpu/Images/TilerParityTests.cs | 57 + .../BindlessImageHeapCapacityTests.cs | 55 + .../Gpu/Pipelines/BoundedFillDetectorTests.cs | 52 + .../Pipelines/ComputeRejectionPolicyTests.cs | 3 + .../Pipelines/ConstantFillDetectorTests.cs | 40 + .../Gpu/Pipelines/DescriptorHeapTests.cs | 49 + .../Pipelines/GraphicsRejectionPolicyTests.cs | 4 +- .../Gpu/Pipelines/PipelineCacheStreamTests.cs | 73 + .../Gpu/Pipelines/PipelineTestSupport.cs | 6 + .../Gpu/Pipelines/ShaderPipelineCacheTests.cs | 139 +- .../Gpu/Pipelines/ShaderPlanningDumpTests.cs | 6 +- .../Gpu/Pipelines/ShaderPrewarmListTests.cs | 56 +- .../Gpu/Pipelines/ShaderProgramCacheTests.cs | 309 ++ .../Gpu/Pipelines/VertexInputResolverTests.cs | 32 + .../Rendering/RenderExecutorComputeTests.cs | 62 +- .../Gpu/Rendering/RenderExecutorDrawTests.cs | 105 +- .../Gpu/Rendering/RenderExecutorStateTests.cs | 61 +- .../Rendering/RenderExecutorTestSupport.cs | 64 +- .../Gpu/Rendering/TargetResolverTests.cs | 148 +- .../Scheduling/PresenterSubmissionTests.cs | 2 + .../Gpu/Scheduling/TickOwnedResourcesTests.cs | 49 + .../Gpu/Vulkan/BitReplicateDeviceTests.cs | 54 + .../Gpu/Vulkan/DeviceAddressShaderTests.cs | 580 +++- .../Vulkan/DriverShadowMemoryProbeTests.cs | 151 + .../Vulkan/Float16ConversionDeviceTests.cs | 181 + .../Gpu/Vulkan/FloatSemanticsDeviceTests.cs | 101 + .../Gpu/Vulkan/GlobalMemoryShaderTests.cs | 42 + .../Gpu/Vulkan/HeadlessVulkan.cs | 71 +- .../Gpu/Vulkan/LayoutComputeRunner.cs | 160 +- .../Gpu/Vulkan/LayoutImageShaderTests.cs | 184 +- .../Gpu/Vulkan/LdsOutOfRangeDeviceTests.cs | 62 + .../Vulkan/MoveRelativeRegisterDeviceTests.cs | 46 + .../NativeHalfConversionProbeDeviceTests.cs | 64 + .../Gpu/Vulkan/PackedD16BufferTests.cs | 88 + .../Gpu/Vulkan/PackedF16LiteralDeviceTests.cs | 65 + .../Gpu/Vulkan/RuntimeBufferStrideTests.cs | 102 + .../Gpu/Vulkan/ScalarAbsoluteDeviceTests.cs | 32 + .../Gpu/Vulkan/ScalarPairBitDeviceTests.cs | 72 + .../Vulkan/SdwaFloatModifierDeviceTests.cs | 70 + .../Vulkan/StaticBufferFormatDeviceTests.cs | 104 + .../StructuredControlFlowDeviceTests.cs | 83 + .../Gpu/Vulkan/TessellationWaveDeviceTests.cs | 252 ++ .../Gpu/Vulkan/Wave64HalfMaskDeviceTests.cs | 61 + .../Gpu/Vulkan/WaveShuffleDeviceTests.cs | 22 +- .../Kernel/KernelAioWaitTests.cs | 26 + .../KernelEventQueueWaiterLifetimeTests.cs | 52 + .../Kernel/KernelMemoryCompatExportsTests.cs | 64 + .../Loader/GuestRedZonePatcherTests.cs | 354 ++ .../Memory/GpuMemory/GuestGpuMemoryTests.cs | 14 + .../Memory/GpuMemory/GuestPageTrackerTests.cs | 64 + .../Memory/HostMemoryQueryRunTests.cs | 53 + .../Network/NetExportsTests.cs | 266 ++ .../Pad/TouchContactIdsTests.cs | 45 + .../PlayGo/PlayGoExportsTests.cs | 117 + .../Pthread/PthreadMutexSemanticsTests.cs | 67 + .../Gen5ShaderTranslatorTests.cs | 29 +- .../SharpEmu.Libs.Tests.csproj | 4 + .../SystemServiceExportsTests.cs | 16 + .../AsyncComputeBindingLayoutTests.cs | 52 + .../VideoOut/HostVideoOptionsTests.cs | 38 + .../VideoOut/PresenterImageBindingTests.cs | 227 +- .../VideoOut/PresenterStartupTests.cs | 48 + .../VideoOut/PresenterUnderTest.cs | 2 +- .../RenderHostDeviceTests.BufferTail.cs | 191 ++ ...RenderHostDeviceTests.CompressedExports.cs | 60 + .../RenderHostDeviceTests.SubgroupLanes.cs | 115 + .../RenderHostDeviceTests.Tessellation.cs | 256 ++ .../VideoOut/RenderHostDeviceTests.cs | 152 +- .../VideoOut/ShaderPrewarmPresenterTests.cs | 6 +- .../VideoOut/VideoOutFlipRequestTests.cs | 15 + .../VideoOut/VideoOutOutputSupportTests.cs | 2 +- .../VulkanPresentEncodeFormatTests.cs | 26 + .../DeviceAddressCodeSizeTests.cs | 92 + .../Gen5BarrierMemoryScopeTests.cs | 149 + .../Gen5DualSourceBlendTests.cs | 70 + .../Gen5DumpedSkeletonTests.cs | 76 + .../Gen5ExecFullAnalysisTests.cs | 56 +- .../Gen5ExecRunTests.cs | 201 ++ .../Gen5Float16ArithmeticTests.cs | 126 +- .../Gen5Int16AluTests.cs | 18 + .../Gen5InterpolationParameterTests.cs | 310 +- .../Gen5LargeDispatcherValidationTests.cs | 252 ++ .../Gen5PixelOutputMappingTests.cs | 20 + .../Gen5PixelReadlaneTests.cs | 54 + .../Gen5PrivateLdsTests.cs | 105 + .../Gen5RenderScaleTests.cs | 255 ++ .../Gen5ScalarAbsoluteTests.cs | 58 +- .../Gen5ScalarLaneTransferTests.cs | 90 + .../Gen5StructuredControlFlowTests.cs | 350 ++ .../Gen5TessellationTests.cs | 230 ++ .../Gen5TypedBufferLoadDecodeTests.cs | 35 + .../Gen5WaveMaskLaneFlagTests.cs | 126 + .../Resources/BindingLayoutTests.cs | 29 +- .../Resources/BufferBoundsCheckTests.cs | 58 + .../BufferCandidateTablePlannerTests.cs | 57 +- .../Resources/BufferFormatLoadTests.cs | 61 + .../Resources/BufferUnalignedWordTests.cs | 46 + .../Resources/DescriptorReadTests.cs | 75 + .../Resources/DirectImageTableTests.cs | 115 +- .../Resources/IndirectSelectorValuesTests.cs | 6 +- .../Resources/PlannedScalarTableLoadTests.cs | 87 + .../ResourceMaterializationCacheTests.cs | 627 ++-- .../Resources/ResourceMaterializerTests.cs | 111 +- .../Resources/ResourceTestProgram.cs | 9 +- .../Resources/ResourceTrackerTests.cs | 158 +- .../RuntimeDescriptorEmissionTests.cs | 189 ++ .../Resources/ScalarValueGraphTests.cs | 28 +- .../Resources/SpirvBindingDeclarationTests.cs | 60 +- .../Resources/WaveMaskSelectorBoundsTests.cs | 3 +- tools/SharpEmu.Tools.ShaderDump/Program.cs | 117 + 384 files changed, 33234 insertions(+), 3476 deletions(-) create mode 100644 LICENSES/GPL-2.0-only.txt create mode 100644 docs/rendering-resolution.md create mode 100644 src/SharpEmu.CLI/RosettaGcBudget.cs create mode 100644 src/SharpEmu.Libs/Gpu/Buffers/GpuBufferAddressBook.cs create mode 100644 src/SharpEmu.Libs/Gpu/Buffers/GuestBufferCache.Watch.cs create mode 100644 src/SharpEmu.Libs/Gpu/Buffers/IDeviceMemoryAllocator.cs create mode 100644 src/SharpEmu.Libs/Gpu/Buffers/OptimalImageMemoryPool.cs create mode 100644 src/SharpEmu.Libs/Gpu/Buffers/SharedPageShadows.cs create mode 100644 src/SharpEmu.Libs/Gpu/Images/CachedImage.Scaling.cs create mode 100644 src/SharpEmu.Libs/Gpu/Images/ImageDropTrace.cs create mode 100644 src/SharpEmu.Libs/Gpu/Images/RenderScalePolicy.cs create mode 100644 src/SharpEmu.Libs/Gpu/Pipelines/BindlessImageHeap.cs create mode 100644 src/SharpEmu.Libs/Gpu/Pipelines/ShaderPipelineCache.Tessellation.cs create mode 100644 src/SharpEmu.Libs/Gpu/Rendering/RenderExecutor.Tessellation.cs create mode 100644 src/SharpEmu.Libs/Gpu/Rendering/TessellationProfile.cs create mode 100644 src/SharpEmu.Libs/Gpu/Scheduling/TickOwnedResources.cs create mode 100644 src/SharpEmu.Libs/Gpu/TessellationOffchip.cs create mode 100644 src/SharpEmu.Libs/Gpu/Vulkan/NativeHalfConversionProbe.cs create mode 100644 src/SharpEmu.Libs/Gpu/Vulkan/ZeroOutOfBoundsReadProbe.cs create mode 100644 src/SharpEmu.Libs/Pad/TouchContactIds.cs create mode 100644 src/SharpEmu.Libs/VideoOut/VulkanFloat16Support.cs create mode 100644 src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.RuntimeDescriptors.cs create mode 100644 src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvTranslator.CooperativeWaves.cs create mode 100644 src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvTranslator.RenderScale.cs create mode 100644 src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvTranslator.StructuredControlFlow.cs create mode 100644 src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvTranslator.Tessellation.cs create mode 100644 src/SharpEmu.ShaderCompiler.Vulkan/Gen5TessellationBridge.cs create mode 100644 src/SharpEmu.ShaderCompiler/Gen5TessellationData.cs create mode 100644 src/SharpEmu.ShaderCompiler/Gen5TessellationInfo.cs create mode 100644 src/SharpEmu.ShaderCompiler/Gen5TessellationLowering.cs create mode 100644 src/SharpEmu.ShaderCompiler/Ir/Gen5ExecRunAnalysis.cs create mode 100644 src/SharpEmu.ShaderCompiler/Resources/RuntimeDescriptorTable.cs create mode 100644 tests/SharpEmu.Libs.Tests/Agc/AgcDirectDrawModifierTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/Agc/AgcEventWriteTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/Agc/AgcMarkerTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/Audio/AudioOut2QueueLevelExportsTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/Cli/RosettaGcBudgetTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/Cpu/GuestThreadRegionAllocationTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/Cpu/ImportShutdownTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/Cpu/PosixSleepStubTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/Gpu/Buffers/DeviceAddressFaultSpanTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/Gpu/Buffers/OptimalImageMemoryPoolTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/Gpu/Buffers/SharedPageShadowsTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/Gpu/Images/MemoryBudgetTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/Gpu/Images/RenderScalePolicyTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/Gpu/Pipelines/BindlessImageHeapCapacityTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/Gpu/Pipelines/ConstantFillDetectorTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/Gpu/Pipelines/PipelineCacheStreamTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/Gpu/Scheduling/TickOwnedResourcesTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/Gpu/Vulkan/BitReplicateDeviceTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/Gpu/Vulkan/DriverShadowMemoryProbeTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/Gpu/Vulkan/Float16ConversionDeviceTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/Gpu/Vulkan/LdsOutOfRangeDeviceTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/Gpu/Vulkan/MoveRelativeRegisterDeviceTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/Gpu/Vulkan/NativeHalfConversionProbeDeviceTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/Gpu/Vulkan/PackedD16BufferTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/Gpu/Vulkan/PackedF16LiteralDeviceTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/Gpu/Vulkan/RuntimeBufferStrideTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/Gpu/Vulkan/ScalarPairBitDeviceTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/Gpu/Vulkan/SdwaFloatModifierDeviceTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/Gpu/Vulkan/StaticBufferFormatDeviceTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/Gpu/Vulkan/StructuredControlFlowDeviceTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/Gpu/Vulkan/TessellationWaveDeviceTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/Memory/HostMemoryQueryRunTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/Pad/TouchContactIdsTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/VideoOut/AsyncComputeBindingLayoutTests.cs create mode 100644 tests/SharpEmu.Libs.Tests/VideoOut/RenderHostDeviceTests.BufferTail.cs create mode 100644 tests/SharpEmu.Libs.Tests/VideoOut/RenderHostDeviceTests.CompressedExports.cs create mode 100644 tests/SharpEmu.Libs.Tests/VideoOut/RenderHostDeviceTests.SubgroupLanes.cs create mode 100644 tests/SharpEmu.Libs.Tests/VideoOut/RenderHostDeviceTests.Tessellation.cs create mode 100644 tests/SharpEmu.ShaderCompiler.Tests/DeviceAddressCodeSizeTests.cs create mode 100644 tests/SharpEmu.ShaderCompiler.Tests/Gen5BarrierMemoryScopeTests.cs create mode 100644 tests/SharpEmu.ShaderCompiler.Tests/Gen5DualSourceBlendTests.cs create mode 100644 tests/SharpEmu.ShaderCompiler.Tests/Gen5DumpedSkeletonTests.cs create mode 100644 tests/SharpEmu.ShaderCompiler.Tests/Gen5ExecRunTests.cs create mode 100644 tests/SharpEmu.ShaderCompiler.Tests/Gen5LargeDispatcherValidationTests.cs create mode 100644 tests/SharpEmu.ShaderCompiler.Tests/Gen5PixelReadlaneTests.cs create mode 100644 tests/SharpEmu.ShaderCompiler.Tests/Gen5PrivateLdsTests.cs create mode 100644 tests/SharpEmu.ShaderCompiler.Tests/Gen5RenderScaleTests.cs create mode 100644 tests/SharpEmu.ShaderCompiler.Tests/Gen5StructuredControlFlowTests.cs create mode 100644 tests/SharpEmu.ShaderCompiler.Tests/Gen5TessellationTests.cs create mode 100644 tests/SharpEmu.ShaderCompiler.Tests/Gen5WaveMaskLaneFlagTests.cs create mode 100644 tests/SharpEmu.ShaderCompiler.Tests/Resources/BufferBoundsCheckTests.cs create mode 100644 tests/SharpEmu.ShaderCompiler.Tests/Resources/BufferFormatLoadTests.cs create mode 100644 tests/SharpEmu.ShaderCompiler.Tests/Resources/BufferUnalignedWordTests.cs create mode 100644 tests/SharpEmu.ShaderCompiler.Tests/Resources/DescriptorReadTests.cs create mode 100644 tests/SharpEmu.ShaderCompiler.Tests/Resources/PlannedScalarTableLoadTests.cs create mode 100644 tests/SharpEmu.ShaderCompiler.Tests/Resources/RuntimeDescriptorEmissionTests.cs diff --git a/LICENSES/GPL-2.0-only.txt b/LICENSES/GPL-2.0-only.txt new file mode 100644 index 0000000..17cb286 --- /dev/null +++ b/LICENSES/GPL-2.0-only.txt @@ -0,0 +1,117 @@ +GNU GENERAL PUBLIC LICENSE +Version 2, June 1991 + +Copyright (C) 1989, 1991 Free Software Foundation, Inc. +51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA + +Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. + +Preamble + +The licenses for most software are designed to take away your freedom to share and change it. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change free software--to make sure the software is free for all its users. This General Public License applies to most of the Free Software Foundation's software and to any other program whose authors commit to using it. (Some other Free Software Foundation software is covered by the GNU Lesser General Public License instead.) You can apply it to your programs, too. + +When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for this service if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs; and that you know you can do these things. + +To protect your rights, we need to make restrictions that forbid anyone to deny you these rights or to ask you to surrender the rights. These restrictions translate to certain responsibilities for you if you distribute copies of the software, or if you modify it. + +For example, if you distribute copies of such a program, whether gratis or for a fee, you must give the recipients all the rights that you have. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. + +We protect your rights with two steps: (1) copyright the software, and (2) offer you this license which gives you legal permission to copy, distribute and/or modify the software. + +Also, for each author's protection and ours, we want to make certain that everyone understands that there is no warranty for this free software. If the software is modified by someone else and passed on, we want its recipients to know that what they have is not the original, so that any problems introduced by others will not reflect on the original authors' reputations. + +Finally, any free program is threatened constantly by software patents. We wish to avoid the danger that redistributors of a free program will individually obtain patent licenses, in effect making the program proprietary. To prevent this, we have made it clear that any patent must be licensed for everyone's free use or not licensed at all. + +The precise terms and conditions for copying, distribution and modification follow. + +TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + +0. This License applies to any program or other work which contains a notice placed by the copyright holder saying it may be distributed under the terms of this General Public License. The "Program", below, refers to any such program or work, and a "work based on the Program" means either the Program or any derivative work under copyright law: that is to say, a work containing the Program or a portion of it, either verbatim or with modifications and/or translated into another language. (Hereinafter, translation is included without limitation in the term "modification".) Each licensee is addressed as "you". + +Activities other than copying, distribution and modification are not covered by this License; they are outside its scope. The act of running the Program is not restricted, and the output from the Program is covered only if its contents constitute a work based on the Program (independent of having been made by running the Program). Whether that is true depends on what the Program does. + +1. You may copy and distribute verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keep intact all the notices that refer to this License and to the absence of any warranty; and give any other recipients of the Program a copy of this License along with the Program. + +You may charge a fee for the physical act of transferring a copy, and you may at your option offer warranty protection in exchange for a fee. + +2. You may modify your copy or copies of the Program or any portion of it, thus forming a work based on the Program, and copy and distribute such modifications or work under the terms of Section 1 above, provided that you also meet all of these conditions: + + a) You must cause the modified files to carry prominent notices stating that you changed the files and the date of any change. + + b) You must cause any work that you distribute or publish, that in whole or in part contains or is derived from the Program or any part thereof, to be licensed as a whole at no charge to all third parties under the terms of this License. + + c) If the modified program normally reads commands interactively when run, you must cause it, when started running for such interactive use in the most ordinary way, to print or display an announcement including an appropriate copyright notice and a notice that there is no warranty (or else, saying that you provide a warranty) and that users may redistribute the program under these conditions, and telling the user how to view a copy of this License. (Exception: if the Program itself is interactive but does not normally print such an announcement, your work based on the Program is not required to print an announcement.) + +These requirements apply to the modified work as a whole. If identifiable sections of that work are not derived from the Program, and can be reasonably considered independent and separate works in themselves, then this License, and its terms, do not apply to those sections when you distribute them as separate works. But when you distribute the same sections as part of a whole which is a work based on the Program, the distribution of the whole must be on the terms of this License, whose permissions for other licensees extend to the entire whole, and thus to each and every part regardless of who wrote it. + +Thus, it is not the intent of this section to claim rights or contest your rights to work written entirely by you; rather, the intent is to exercise the right to control the distribution of derivative or collective works based on the Program. + +In addition, mere aggregation of another work not based on the Program with the Program (or with a work based on the Program) on a volume of a storage or distribution medium does not bring the other work under the scope of this License. + +3. You may copy and distribute the Program (or a work based on it, under Section 2) in object code or executable form under the terms of Sections 1 and 2 above provided that you also do one of the following: + + a) Accompany it with the complete corresponding machine-readable source code, which must be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, + + b) Accompany it with a written offer, valid for at least three years, to give any third party, for a charge no more than your cost of physically performing source distribution, a complete machine-readable copy of the corresponding source code, to be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, + + c) Accompany it with the information you received as to the offer to distribute corresponding source code. (This alternative is allowed only for noncommercial distribution and only if you received the program in object code or executable form with such an offer, in accord with Subsection b above.) + +The source code for a work means the preferred form of the work for making modifications to it. For an executable work, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the executable. However, as a special exception, the source code distributed need not include anything that is normally distributed (in either source or binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that component itself accompanies the executable. + +If distribution of executable or object code is made by offering access to copy from a designated place, then offering equivalent access to copy the source code from the same place counts as distribution of the source code, even though third parties are not compelled to copy the source along with the object code. + +4. You may not copy, modify, sublicense, or distribute the Program except as expressly provided under this License. Any attempt otherwise to copy, modify, sublicense or distribute the Program is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance. + +5. You are not required to accept this License, since you have not signed it. However, nothing else grants you permission to modify or distribute the Program or its derivative works. These actions are prohibited by law if you do not accept this License. Therefore, by modifying or distributing the Program (or any work based on the Program), you indicate your acceptance of this License to do so, and all its terms and conditions for copying, distributing or modifying the Program or works based on it. + +6. Each time you redistribute the Program (or any work based on the Program), the recipient automatically receives a license from the original licensor to copy, distribute or modify the Program subject to these terms and conditions. You may not impose any further restrictions on the recipients' exercise of the rights granted herein. You are not responsible for enforcing compliance by third parties to this License. + +7. If, as a consequence of a court judgment or allegation of patent infringement or for any other reason (not limited to patent issues), conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot distribute so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not distribute the Program at all. For example, if a patent license would not permit royalty-free redistribution of the Program by all those who receive copies directly or indirectly through you, then the only way you could satisfy both it and this License would be to refrain entirely from distribution of the Program. + +If any portion of this section is held invalid or unenforceable under any particular circumstance, the balance of the section is intended to apply and the section as a whole is intended to apply in other circumstances. + +It is not the purpose of this section to induce you to infringe any patents or other property right claims or to contest validity of any such claims; this section has the sole purpose of protecting the integrity of the free software distribution system, which is implemented by public license practices. Many people have made generous contributions to the wide range of software distributed through that system in reliance on consistent application of that system; it is up to the author/donor to decide if he or she is willing to distribute software through any other system and a licensee cannot impose that choice. + +This section is intended to make thoroughly clear what is believed to be a consequence of the rest of this License. + +8. If the distribution and/or use of the Program is restricted in certain countries either by patents or by copyrighted interfaces, the original copyright holder who places the Program under this License may add an explicit geographical distribution limitation excluding those countries, so that distribution is permitted only in or among countries not thus excluded. In such case, this License incorporates the limitation as if written in the body of this License. + +9. The Free Software Foundation may publish revised and/or new versions of the General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. + +Each version is given a distinguishing version number. If the Program specifies a version number of this License which applies to it and "any later version", you have the option of following the terms and conditions either of that version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of this License, you may choose any version ever published by the Free Software Foundation. + +10. If you wish to incorporate parts of the Program into other free programs whose distribution conditions are different, write to the author to ask for permission. For software which is copyrighted by the Free Software Foundation, write to the Free Software Foundation; we sometimes make exceptions for this. Our decision will be guided by the two goals of preserving the free status of all derivatives of our free software and of promoting the sharing and reuse of software generally. + +NO WARRANTY + +11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + +12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. + +END OF TERMS AND CONDITIONS + +How to Apply These Terms to Your New Programs + +If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. + +To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively convey the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. + + one line to give the program's name and an idea of what it does. Copyright (C) yyyy name of author + + This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. + + You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. Also add information on how to contact you by electronic and paper mail. + +If the program is interactive, make it output a short notice like this when it starts in an interactive mode: + + Gnomovision version 69, Copyright (C) year name of author Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, the commands you use may be called something other than `show w' and `show c'; they could even be mouse-clicks or menu items--whatever suits your program. + +You should also get your employer (if you work as a programmer) or your school, if any, to sign a "copyright disclaimer" for the program, if necessary. Here is a sample; alter the names: + + Yoyodyne, Inc., hereby disclaims all copyright interest in the program `Gnomovision' (which makes passes at compilers) written by James Hacker. + +signature of Ty Coon, 1 April 1989 Ty Coon, President of Vice diff --git a/docs/rendering-resolution.md b/docs/rendering-resolution.md new file mode 100644 index 0000000..6a41d18 --- /dev/null +++ b/docs/rendering-resolution.md @@ -0,0 +1,112 @@ + + +# Rendering resolution + +Options → Rendering exposes three independent sizes, globally and per game: + +- **Resolution** sets the host window size or exclusive fullscreen mode. +- **Game display mode** reports a 1080p display to the game by default. Select + 3840×2160 to request a 4K display. A registered presentation buffer cannot + change this setting. +- **Internal resolution scale** multiplies the host size of the game's + screen-sized render targets. 100% is native and changes nothing. + +The CLI equivalents are `--resolution=1280x720`, +`--guest-resolution=1920x1080` (or `3840x2160`) and +`--render-scale=0.5`. `SHARPEMU_RENDER_SCALE=0.5` does the same for scripted +runs and overrides the stored setting. The accepted range is 0.25 to 2, +rounded to a hundredth. + +The game chooses its internal render targets. Some games adapt those targets to +the reported display mode; others render at a fixed resolution and may still +register 4K buffers with a 1080p display. Silent Hill: The Short Message does so +in current testing. The size of that presentation buffer alone also does not +establish the sizes of its intermediate scene targets. Changing the window alone +does not reduce the work the game renders; the internal resolution scale does. + +## How the scale is applied + +Guest memory layouts never change. A scaled image keeps its guest geometry in +its description - extent, pitch, tile mode, mip layout - and only its host +backing is created at `ceil(guest × scale)`. Everything that would notice: + +- **Rasterization.** The render area, the viewport and the scissor are scaled + with the attachments. Scissor edges round outward so no guest pixel loses its + host coverage. With clipping disabled the vertex program bakes the viewport + transform against a reference extent that is divided by an upscale, so the + same program maps guest pixels onto host positions at either resolution. +- **Shaders.** `gl_FragCoord` is divided by the attachments' factor as it is + loaded, so the pixel position inputs and everything derived from them - screen + UVs above all - stay in guest pixels. Integer image coordinates are mapped + onto host texels through the bound image's factor, and resource-info queries + report guest sizes. Dynamic sample offsets are converted from guest texels + to host texels before normalization. The factors and a mask of scaled image resources travel in + shader data, so no shader is specialized per scale. +- **Guest transfers.** An upload fills a guest-resolution twin of the image from + the copy regions the tiler built and blits into the backing; a download blits + the backing into the twin and copies out of it. A copy between two images that + disagree on host resolution runs at guest resolution the same way. +- **Presentation.** The flip path already works from the host extent of the + captured image, so a scaled flip buffer still fills the window. + +## What scales and what does not + +Only screen-class, single-level, single-sample, uncompressed 2D render and depth +targets scale: width ≥ 1280, height ≥ 720, not square, render or depth tiling, +guest-placed, in a format the device can blit. The decision is a pure function +of the guest image description, so every view, alias and copy of one guest range +agrees on it. + +Everything else keeps its guest resolution: + +- block-compressed images, multisample footprints, mip chains, cube and array + faces, 3D images; +- depth targets with a stencil plane, because Vulkan cannot blit stencil and the + guest-resolution twin could not be filled; +- HTile, which encodes guest depth blocks the renderer reads back as clear + state. DCC only describes guest bytes the host never stores, so it does scale; +- asset tilings (linear, standard), shadow maps, lookup tables, probe atlases + and eye-adaptation readbacks, which are below the size threshold or square; +- storage images the moment a compute program writes one. The guest dispatch + grid does not shrink with the host image, so a scaled image would be written + with holes above 100% and redundantly below. The guest range is remembered and + every later image over that memory stays native; +- every attachment of a pass whose attachments disagree, which happens when one + of them falls into a case above. The draw drops the scaled attachments back to + guest resolution and resolves its targets again. + +Only the Vulkan backend scales; the Metal backend ignores the setting. + +## Known limitations + +- **Compute-written surfaces never scale.** In a deferred renderer that is a + large part of the frame, so the saving is smaller than the scale suggests. +- **Depth with stencil never scales**, and it takes its pass with it. +- **Tone and screen-space effects shift.** Measured on Silent Hill: The Short + Message, the title scene is correct in composition, coverage and UI placement + at 50% and 200%, but slightly brighter than native at both, and at 50% the + character's coat picks up colour speckle. Effects that reason about texels + rather than about the screen - screen-space shadows, subsurface scattering, + dithering - change their footprint with the host resolution. +- **Guest stencil uploads and downloads** of a scaled image would lose the + stencil plane; this cannot happen today only because stencil images never + scale. + +## Measured effect + +Silent Hill: The Short Message at the title menu, 1080p display mode, GPU time +per draw scope (averaged over a whole run): + +| scope | 50% | 100% | 200% | +|---|---|---|---| +| `vs=CBE40391 ps=E0D7BA42` | 1.68 ms | 7.60 ms | 41.81 ms | +| `vs=9A9C9C47 ps=B2FD2C67` | 0.71 ms | 2.44 ms | 10.52 ms | +| `vs=9A9C9C47 ps=E5A1AB7F` | 0.78 ms | 2.35 ms | 8.27 ms | + +Pixel work follows the area, as it should. Frame rate does not: this title is +bound by guest command translation and compute, not by pixels, and stays at +roughly 2.5 fps at every scale. Dreaming Sarah holds 60 fps at 50% with one +scaled target. diff --git a/src/SharpEmu.CLI/Program.cs b/src/SharpEmu.CLI/Program.cs index 197e9f3..a2daec2 100644 --- a/src/SharpEmu.CLI/Program.cs +++ b/src/SharpEmu.CLI/Program.cs @@ -9,6 +9,7 @@ using SharpEmu.Libs.VideoOut; using SharpEmu.Logging; using System.Runtime.InteropServices; using System.Runtime.Loader; +using System.Globalization; using System.Text; using System.Text.Json; @@ -45,6 +46,7 @@ internal static partial class Program [STAThread] private static int Main(string[] args) { + RosettaGcBudget.ReexecIfNeeded(args); ConfigureManagedPluginResolution(); if (args.Length > 0 && args[0] == SharpEmu.Core.Diagnostics.WindowsCrashCapture.HelperArgument) @@ -1010,7 +1012,7 @@ internal static partial class Program private static void PrintUsage() { - Log.Info("Usage: SharpEmu.CLI [--strict] [--trace-imports[=N]] [--cpu-engine=] [--console=] [--console-language=] [--log-level=] [--log-file[=]] [--window-mode=] [--resolution=] [--display=] [--refresh-rate=] [--scaling=] [--vsync=] [--hdr=] [--overlay=] [--overlay-mode=] [--overlay-corner=] [--debug-server[=host:port]] "); + Log.Info("Usage: SharpEmu.CLI [--strict] [--trace-imports[=N]] [--cpu-engine=] [--console=] [--console-language=] [--log-level=] [--log-file[=]] [--window-mode=] [--resolution=] [--guest-resolution=<1920x1080|3840x2160>] [--render-scale=<0.5|0.75|1|1.5|2>] [--display=] [--refresh-rate=] [--scaling=] [--vsync=] [--hdr=] [--overlay=] [--overlay-mode=] [--overlay-corner=] [--debug-server[=host:port]] "); Log.Info(@"Example: SharpEmu.CLI --cpu-engine=native --trace-imports=64 --log-level=debug --log-file ""E:\Games\...\eboot.bin"""); Log.Info("Debug server: --debug-server starts a live debug listener (default 127.0.0.1:5714); connect with SharpEmu.DebugClient."); } @@ -1075,6 +1077,8 @@ internal static partial class Program var systemLanguage = 1; HostWindowMode? windowModeOverride = null; HostScalingMode? scalingModeOverride = null; + GuestDisplayResolution? guestResolutionOverride = null; + float? renderScaleOverride = null; int? windowWidthOverride = null; int? windowHeightOverride = null; int? displayIndexOverride = null; @@ -1134,6 +1138,31 @@ internal static partial class Program windowHeightOverride = windowHeight; continue; } + if (TrySplitOption(argument, "--guest-resolution", out var guestResolutionText)) + { + if (guestResolutionText is not ("1920x1080" or "3840x2160")) + { + ebootPath = string.Empty; + runtimeOptions = default; + return false; + } + guestResolutionOverride = guestResolutionText == "3840x2160" + ? GuestDisplayResolution.UltraHd : GuestDisplayResolution.Hd; + continue; + } + if (TrySplitOption(argument, "--render-scale", out var renderScaleText)) + { + if (!float.TryParse(renderScaleText, NumberStyles.Float, CultureInfo.InvariantCulture, out var renderScale) || + !float.IsFinite(renderScale) || + renderScale < HostVideoOptions.MinimumRenderScale || renderScale > HostVideoOptions.MaximumRenderScale) + { + ebootPath = string.Empty; + runtimeOptions = default; + return false; + } + renderScaleOverride = renderScale; + continue; + } if (TrySplitOption(argument, "--display", out var displayText)) { if (!int.TryParse(displayText, out var displayIndex) || displayIndex < 0) @@ -1391,6 +1420,8 @@ internal static partial class Program { WindowMode = windowModeOverride ?? configuredVideoOptions.WindowMode, ScalingMode = scalingModeOverride ?? configuredVideoOptions.ScalingMode, + GuestResolution = guestResolutionOverride ?? configuredVideoOptions.GuestResolution, + RenderScale = renderScaleOverride ?? configuredVideoOptions.RenderScale, Width = windowWidthOverride ?? configuredVideoOptions.Width, Height = windowHeightOverride ?? configuredVideoOptions.Height, DisplayIndex = displayIndexOverride ?? configuredVideoOptions.DisplayIndex, @@ -1431,6 +1462,9 @@ internal static partial class Program { WindowMode = windowMode, ScalingMode = scalingMode, + GuestResolution = effective.GuestResolution == "3840x2160" + ? GuestDisplayResolution.UltraHd : GuestDisplayResolution.Hd, + RenderScale = effective.RenderScale, Width = hasResolution ? configuredWidth : defaults.Width, Height = hasResolution ? configuredHeight : defaults.Height, DisplayIndex = effective.DisplayIndex, diff --git a/src/SharpEmu.CLI/RosettaGcBudget.cs b/src/SharpEmu.CLI/RosettaGcBudget.cs new file mode 100644 index 0000000..6ad3af2 --- /dev/null +++ b/src/SharpEmu.CLI/RosettaGcBudget.cs @@ -0,0 +1,100 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Collections; +using System.Runtime.InteropServices; + +namespace SharpEmu.CLI; + +/// +/// Under Rosetta 2 every garbage collection suspends the runtime with +/// FlushProcessWriteBuffers, which reads each thread's registers through a Mach call that +/// Rosetta services by parking the thread in its runtime. A signal that reaches a parked +/// thread (guest memory tracking raises thousands per second) trips Rosetta's "expected saved +/// LR to be in translated code" assertion and wedges the process. With the default gen0 budget +/// the emulator collects about twice a second; a 64 MiB budget makes that rare. +/// +/// +/// The GC reads its gen0 budget only from the environment when the runtime starts, so the +/// process re-executes itself once with the variable set. execve keeps the process id. +/// +internal static class RosettaGcBudget +{ + private const string BudgetVariable = "DOTNET_GCgen0size"; + // 64 MiB: a larger budget turns every collection into a blocking gen1/gen2 pass of + // about a second (512 MiB measured 8 such pauses in 150 s of Demon's Souls). + private const string Budget = "0x4000000"; + private const string OptOutVariable = "SHARPEMU_ROSETTA_GC_BUDGET"; + + public static void ReexecIfNeeded(string[] args) + { + if (!OperatingSystem.IsMacOS() || + RuntimeInformation.ProcessArchitecture != Architecture.X64 || + Environment.GetEnvironmentVariable(BudgetVariable) is not null || + Environment.GetEnvironmentVariable(OptOutVariable) == "0" || + !IsTranslated()) + { + return; + } + + var path = Environment.ProcessPath; + if (string.IsNullOrEmpty(path)) + { + return; + } + + var environment = new List(); + foreach (DictionaryEntry entry in Environment.GetEnvironmentVariables()) + { + environment.Add($"{entry.Key}={entry.Value}"); + } + environment.Add($"{BudgetVariable}={Budget}"); + + var argv = BuildArguments(path, Environment.GetCommandLineArgs()[0], args); + var nativeArguments = ToNullTerminated(argv); + var nativeEnvironment = ToNullTerminated(environment); + // Only returns on failure; the emulator then runs with the default budget. + _ = Execve(path, nativeArguments, nativeEnvironment); + var error = Marshal.GetLastPInvokeError(); + foreach (var pointer in nativeArguments) Marshal.FreeCoTaskMem(pointer); + foreach (var pointer in nativeEnvironment) Marshal.FreeCoTaskMem(pointer); + Console.Error.WriteLine( + $"[LOADER][WARN] Could not restart with a larger GC budget under Rosetta: errno={error}"); + } + + internal static string[] BuildArguments(string executable, string entryAssembly, string[] args) => + Path.GetFileNameWithoutExtension(executable).Equals("dotnet", StringComparison.OrdinalIgnoreCase) + ? [executable, entryAssembly, .. args] + : [executable, .. args]; + + private static bool IsTranslated() + { + var value = 0; + var size = (nint)sizeof(int); + return SysctlByName("sysctl.proc_translated", ref value, ref size, 0, 0) == 0 && value == 1; + } + + private static nint[] ToNullTerminated(IReadOnlyList values) + { + var pointers = new nint[values.Count + 1]; + for (var i = 0; i < values.Count; i++) + { + pointers[i] = Marshal.StringToCoTaskMemUTF8(values[i]); + } + return pointers; + } + + [DllImport("libc", EntryPoint = "execve", SetLastError = true)] + private static extern int Execve( + [MarshalAs(UnmanagedType.LPUTF8Str)] string path, + nint[] argv, + nint[] envp); + + [DllImport("libc", EntryPoint = "sysctlbyname")] + private static extern int SysctlByName( + [MarshalAs(UnmanagedType.LPUTF8Str)] string name, + ref int value, + ref nint size, + nint newValue, + nint newSize); +} diff --git a/src/SharpEmu.Core/Cpu/Native/DirectExecutionBackend.FastPath.cs b/src/SharpEmu.Core/Cpu/Native/DirectExecutionBackend.FastPath.cs index 433464e..feefaf8 100644 --- a/src/SharpEmu.Core/Cpu/Native/DirectExecutionBackend.FastPath.cs +++ b/src/SharpEmu.Core/Cpu/Native/DirectExecutionBackend.FastPath.cs @@ -261,23 +261,40 @@ public sealed unsafe partial class DirectExecutionBackend return code; } + // SHARPEMU_MEMCPY_MANAGED_MIN overrides the size from which guest memcpy takes the managed path. + internal static readonly int ManagedCopyMinimumBytes = + int.TryParse(Environment.GetEnvironmentVariable("SHARPEMU_MEMCPY_MANAGED_MIN"), out var minimum) && minimum > 0 ? minimum : 4096; + /// /// memcpy/memmove(dst, src, n) as a forward rep movsb, returning dst. The libc /// export copies with memmove semantics, so a destination that starts inside the source /// (where a forward copy would read bytes it already overwrote) takes the managed path. + /// So does a copy of at least : the native copy takes + /// one write-tracking fault per GPU-watched page it writes, a signal round trip of about + /// 0.2 ms under Rosetta, while the managed copy marks the whole destination written first. /// Only RAX, RCX, RSI, RDI and the flags change - all caller-saved - and the direction /// flag is clear on entry per the SysV ABI. /// internal static List EmitGuestMemoryCopyStub(nint fallbackTrampoline) { - var code = new List(40); + var code = new List(48); + var slowPathFixups = new List(); // mov rcx, rdi / sub rcx, rsi / cmp rcx, rdx / jb slow: dst - src < n (unsigned) is // exactly a destination in (src, src + n) or equal to src with a non-empty copy. code.AddRange([0x48, 0x89, 0xF9, 0x48, 0x29, 0xF1, 0x48, 0x39, 0xD1, 0x72, 0x00]); - var slowPathFixup = code.Count - 1; + slowPathFixups.Add(code.Count - 1); + // cmp rdx, imm32 / jae slow + code.AddRange([0x48, 0x81, 0xFA]); + code.AddRange(BitConverter.GetBytes(ManagedCopyMinimumBytes)); + code.AddRange([0x73, 0x00]); + slowPathFixups.Add(code.Count - 1); // mov rax, rdi / mov rcx, rdx / rep movsb / ret code.AddRange([0x48, 0x89, 0xF8, 0x48, 0x89, 0xD1, 0xF3, 0xA4, 0xC3]); - code[slowPathFixup] = checked((byte)(code.Count - (slowPathFixup + 1))); + foreach (var fixup in slowPathFixups) + { + code[fixup] = checked((byte)(code.Count - (fixup + 1))); + } + // jmp qword [rip+0] with the absolute trampoline address after it. code.AddRange([0xFF, 0x25, 0x00, 0x00, 0x00, 0x00]); code.AddRange(BitConverter.GetBytes((long)fallbackTrampoline)); diff --git a/src/SharpEmu.Core/Cpu/Native/DirectExecutionBackend.Imports.cs b/src/SharpEmu.Core/Cpu/Native/DirectExecutionBackend.Imports.cs index 07f19b2..965452a 100644 --- a/src/SharpEmu.Core/Cpu/Native/DirectExecutionBackend.Imports.cs +++ b/src/SharpEmu.Core/Cpu/Native/DirectExecutionBackend.Imports.cs @@ -36,6 +36,7 @@ public sealed partial class DirectExecutionBackend private readonly object _importResultLogSampleGate = new(); private readonly Dictionary _importResultLogSamples = new(StringComparer.Ordinal); + private readonly Dictionary _unresolvedImportLogSamples = new(StringComparer.Ordinal); private int _il2CppExceptionDiagnosticCount; private static ulong ImportDispatchGatewayManaged(nint backendHandle, int importIndex, nint argPackPtr) @@ -163,6 +164,14 @@ public sealed partial class DirectExecutionBackend LastError = $"Import dispatch index out of range: {importIndex}"; return 18446744071562199042uL; } + // Stop before the leaf fast paths too: services may already be tearing down. + // ActiveForcedGuestExit includes the global request on this backend, so testing + // its negation here would prevent every shutdown redirect. + if (_forcedGuestExit && TryEndGuestSliceForShutdown(argPackPtr)) + { + cpuContext[CpuRegister.Rax] = 1uL; + return 1uL; + } ImportStubEntry importStubEntry = _importEntries[importIndex]; using var registerPacketImport = SharpEmu.Libs.Diagnostics.AgcRegisterPacketProfile.MeasureImport(importStubEntry.Nid); if (_perfHleHistogram) @@ -345,14 +354,6 @@ public sealed partial class DirectExecutionBackend Console.Error.WriteLine( $"[LOADER][TRACE] bootstrap_call#{num}: op=0x{value:X16} sym_ptr=0x{value2:X16} sym='{symbolText}' out_ptr=0x{num3:X16} ret=0x{num7:X16}"); } - // Once the host is shutting down, a guest thread still running reaches no more HLE: the - // services behind the imports are being torn down, and a title that sees them fail (a null - // command buffer from the GPU library, say) crashes the process on its way out. - if (_forcedGuestExit && !ActiveForcedGuestExit && TryEndGuestSliceForShutdown(argPackPtr)) - { - cpuContext[CpuRegister.Rax] = 1uL; - return 1uL; - } if (!isGuestWorker && !ActiveForcedGuestExit && ShouldForceGuestExitOnImportLoop(in importStubEntry, num7, num, value, value2)) @@ -594,9 +595,12 @@ public sealed partial class DirectExecutionBackend { DumpIl2CppExceptionDiagnostic(cpuContext, value, num7); } - Console.Error.WriteLine( - $"[LOADER][WARN] Import#{num} unresolved: nid={importStubEntry.Nid} ret=0x{num7:X16} " + - $"rdi=0x{value:X16} rsi=0x{value2:X16} rdx=0x{num3:X16} rcx=0x{num4:X16} r8=0x{num5:X16} r9=0x{num6:X16}"); + if (ShouldLogUnresolvedImport(importStubEntry.Nid, out var unresolvedCount)) + { + Console.Error.WriteLine( + $"[LOADER][WARN] Import#{num} unresolved occurrence={unresolvedCount}: nid={importStubEntry.Nid} ret=0x{num7:X16} " + + $"rdi=0x{value:X16} rsi=0x{value2:X16} rdx=0x{num3:X16} rcx=0x{num4:X16} r8=0x{num5:X16} r9=0x{num6:X16}"); + } if (importStubEntry.Nid == "L-Q3LEjIbgA") { string value18 = string.Join(" ", importStubEntry.Nid.Select(delegate (char c) @@ -1665,12 +1669,19 @@ public sealed partial class DirectExecutionBackend var expectedUserServiceNoEvent = string.Equals(nid, "yH17Q6NWtVg", StringComparison.Ordinal) && resultValue == unchecked((int)0x80960007); + var expectedSystemServiceNoEvent = + string.Equals(nid, "656LMQSrg6U", StringComparison.Ordinal) && + resultValue == unchecked((int)0x80A10004); var expectedPrivacyInvalidParameter = string.Equals(nid, "D-CzAxQL0XI", StringComparison.Ordinal) && resultValue == unchecked((int)0x80960009); var expectedPlayGoChunkEnumerationEnd = string.Equals(nid, "uWIYLFkkwqk", StringComparison.Ordinal) && resultValue == unchecked((int)0x80B2000C); + // NO_ENTITLEMENT: the title asks about add-on content the user does not own. + var expectedAddcontNotOwned = + string.Equals(nid, "xddD23+8TfQ", StringComparison.Ordinal) && + resultValue == unchecked((int)0x817D0007); if (!expectedFileProbeMiss && !expectedTimedWaitTimeout && !expectedEqueueTimeout && @@ -1682,8 +1693,10 @@ public sealed partial class DirectExecutionBackend !expectedPollSemaBusy && !expectedNetAcceptWouldBlock && !expectedUserServiceNoEvent && + !expectedSystemServiceNoEvent && !expectedPrivacyInvalidParameter && - !expectedPlayGoChunkEnumerationEnd) + !expectedPlayGoChunkEnumerationEnd && + !expectedAddcontNotOwned) { return true; } @@ -1705,6 +1718,19 @@ public sealed partial class DirectExecutionBackend return count <= 8 || count % 10000 == 0; } + private bool ShouldLogUnresolvedImport(string nid, out long count) + { + // Keep the first call sites and periodic totals without formatting a warning + // on every invocation of an unsupported export. Dispatch still returns its error. + lock (_importResultLogSampleGate) + { + _unresolvedImportLogSamples.TryGetValue(nid, out count); + count++; + _unresolvedImportLogSamples[nid] = count; + } + return count <= 8 || count % 10000 == 0; + } + private static bool ShouldLogExpectedImportResults() => string.Equals( Environment.GetEnvironmentVariable("SHARPEMU_LOG_EXPECTED_IMPORT_RESULTS"), @@ -1869,7 +1895,7 @@ public sealed partial class DirectExecutionBackend } } - // Returns the import straight to the host entry stub, which ends this guest slice. + // Return directly to the host entry stub instead of executing another HLE call. private unsafe bool TryEndGuestSliceForShutdown(nint argPackPtr) { ulong sentinel = ActiveEntryReturnSentinelRip; diff --git a/src/SharpEmu.Core/Cpu/Native/DirectExecutionBackend.cs b/src/SharpEmu.Core/Cpu/Native/DirectExecutionBackend.cs index 331b974..e051715 100644 --- a/src/SharpEmu.Core/Cpu/Native/DirectExecutionBackend.cs +++ b/src/SharpEmu.Core/Cpu/Native/DirectExecutionBackend.cs @@ -3982,6 +3982,11 @@ public sealed unsafe partial class DirectExecutionBackend : INativeCpuBackend, I public bool SupportsGuestContextTransfer => true; + // The handle and context this host thread registered last; a repeat registration (every + // scePthreadSelf on an external guest thread, tens of thousands a second) then skips the + // scheduler gate. + [ThreadStatic] private static ulong _registeredExternalHandle; + [ThreadStatic] private static CpuContext? _registeredExternalContext; public bool TryGetGuestThreadStackBounds( ulong threadHandle, out ulong stackBase, @@ -4016,6 +4021,12 @@ public sealed unsafe partial class DirectExecutionBackend : INativeCpuBackend, I return; } + if (_registeredExternalHandle == threadHandle && ReferenceEquals(_registeredExternalContext, context)) + { + Volatile.Write(ref _currentExternalGuestThreadHandle, threadHandle); + return; + } + lock (_guestThreadGate) { _currentExternalGuestThreadHandle = threadHandle; @@ -4029,15 +4040,19 @@ public sealed unsafe partial class DirectExecutionBackend : INativeCpuBackend, I { existing.Context = context; Volatile.Write(ref existing.HostThreadId, hostThreadId); - return; + } + else + { + _externalGuestThreads[threadHandle] = new ExternalGuestThreadState + { + Context = context, + Name = $"External-{threadHandle:X}", + HostThreadId = hostThreadId, + }; } - _externalGuestThreads[threadHandle] = new ExternalGuestThreadState - { - Context = context, - Name = $"External-{threadHandle:X}", - HostThreadId = hostThreadId, - }; + _registeredExternalHandle = threadHandle; + _registeredExternalContext = context; } } @@ -5604,7 +5619,25 @@ public sealed unsafe partial class DirectExecutionBackend : INativeCpuBackend, I return false; } - private static bool TryMapGuestThreadRegion( + // Finding a free slot and mapping it must be one step: Map reuses a region that + // already exists, so two threads claiming at once would both get the same stack. + private static readonly object GuestThreadRegionGate = new(); + + internal static bool TryMapGuestThreadRegion( + IVirtualMemory virtualMemory, + ulong baseAddress, + ulong size, + ProgramHeaderFlags protection, + out ulong mappedBase, + out string? error) + { + lock (GuestThreadRegionGate) + { + return TryMapFreeGuestThreadRegion(virtualMemory, baseAddress, size, protection, out mappedBase, out error); + } + } + + private static bool TryMapFreeGuestThreadRegion( IVirtualMemory virtualMemory, ulong baseAddress, ulong size, @@ -5645,6 +5678,17 @@ public sealed unsafe partial class DirectExecutionBackend : INativeCpuBackend, I IVirtualMemory virtualMemory, out ulong tlsBase, out string? error) + { + lock (GuestThreadRegionGate) + { + return TryMapFreeGuestThreadTlsRegion(virtualMemory, out tlsBase, out error); + } + } + + private static bool TryMapFreeGuestThreadTlsRegion( + IVirtualMemory virtualMemory, + out ulong tlsBase, + out string? error) { for (int i = 0; i < GuestThreadRegionSlots; i++) { @@ -6383,19 +6427,15 @@ public sealed unsafe partial class DirectExecutionBackend : INativeCpuBackend, I return GuestNativeCallExitReason.Exception; } const uint stubSize = 512u; - void* ptr = VirtualAlloc(null, stubSize, 12288u, 4u); - if (ptr == null) + // The stub page and host-RSP slot are kept per host thread and nesting depth: allocating + // and releasing them on every resume cost an mmap/munmap pair and two acquisitions of the + // host-memory gate, which the GPU write tracker holds across its own protection changes. + var depth = _continuationDepth; + if (!TryAcquireContinuationStub(depth, stubSize, out var ptr, out var hostRspStorage, out reason)) { - reason = "failed to allocate executable memory for guest thread stub"; - return GuestNativeCallExitReason.Exception; - } - void* hostRspStorage = NativeMemory.AllocZeroed((nuint)HostStackStateBytes); - if (hostRspStorage == null) - { - VirtualFree(ptr, 0u, 32768u); - reason = "failed to allocate writable host-RSP storage for guest continuation stub"; return GuestNativeCallExitReason.Exception; } + _continuationDepth = depth + 1; var previousActiveBackend = _activeExecutionBackend; var previousActiveContext = _activeCpuContext; var previousSentinel = _activeEntryReturnSentinelRip; @@ -6469,8 +6509,7 @@ public sealed unsafe partial class DirectExecutionBackend : INativeCpuBackend, I reason = $"failed to patch guest continuation return slot at 0x{returnSlotAddress:X16}"; return GuestNativeCallExitReason.Exception; } - uint oldProtect = default(uint); - if (!VirtualProtect(ptr, stubSize, 32u, &oldProtect)) + if (!ProtectContinuationStub(ptr, stubSize, executable: true)) { reason = "failed to seal guest continuation stub execute-read"; return GuestNativeCallExitReason.Exception; @@ -6530,11 +6569,85 @@ public sealed unsafe partial class DirectExecutionBackend : INativeCpuBackend, I previousForcedExit, previousYieldRequested, previousYieldReason); - NativeMemory.Free(hostRspStorage); - VirtualFree(ptr, 0u, 32768u); + _continuationDepth = depth; } } + [ThreadStatic] private static nint[]? _continuationStubPages; + [ThreadStatic] private static nint[]? _continuationHostRspSlots; + [ThreadStatic] private static int _continuationDepth; + + // Returns this thread's stub page for the given nesting depth, writable and with a zeroed + // host-RSP slot; the first use at a depth allocates them, and they live as long as the thread. + private unsafe bool TryAcquireContinuationStub(int depth, uint stubSize, out void* stub, out void* hostRspStorage, out string? reason) + { + reason = null; + stub = null; + hostRspStorage = null; + var pages = _continuationStubPages ??= new nint[4]; + var slots = _continuationHostRspSlots ??= new nint[4]; + if (depth >= pages.Length) + { + Array.Resize(ref pages, depth + 2); + Array.Resize(ref slots, depth + 2); + _continuationStubPages = pages; + _continuationHostRspSlots = slots; + } + + if (pages[depth] == 0) + { + var page = VirtualAlloc(null, stubSize, 12288u, 4u); + if (page == null) + { + reason = "failed to allocate executable memory for guest thread stub"; + return false; + } + + var slot = NativeMemory.AllocZeroed((nuint)HostStackStateBytes); + if (slot == null) + { + VirtualFree(page, 0u, 32768u); + reason = "failed to allocate writable host-RSP storage for guest continuation stub"; + return false; + } + + pages[depth] = (nint)page; + slots[depth] = (nint)slot; + } + else + { + if (!ProtectContinuationStub((void*)pages[depth], stubSize, executable: false)) + { + reason = "failed to reopen guest continuation stub for writing"; + return false; + } + + NativeMemory.Clear((void*)slots[depth], (nuint)HostStackStateBytes); + } + + stub = (void*)pages[depth]; + hostRspStorage = (void*)slots[depth]; + return true; + } + + // The stub page is emulator-private, so on POSIX its protection changes bypass the tracked + // host-memory layer and its gate. + private static unsafe bool ProtectContinuationStub(void* stub, uint stubSize, bool executable) + { + if (OperatingSystem.IsWindows()) + { + uint oldProtect = 0; + return VirtualProtect(stub, stubSize, executable ? 32u : 4u, &oldProtect); + } + + var pageSize = (nint)Environment.SystemPageSize; + var page = (nint)stub & ~(pageSize - 1); + return PosixMprotect(page, (nuint)pageSize, executable ? 1 | 4 : 1 | 2) == 0; + } + + [DllImport("libc", EntryPoint = "mprotect")] + private static extern int PosixMprotect(nint address, nuint length, int protection); + // The continuation trampoline is rebuilt on every blocked-thread resume. // Keep its tiny writer on the stack: capturing local emit functions create a // managed display-class allocation on this extremely hot path. diff --git a/src/SharpEmu.Core/Cpu/Native/PosixHostStubs.cs b/src/SharpEmu.Core/Cpu/Native/PosixHostStubs.cs index 99726f2..8005aeb 100644 --- a/src/SharpEmu.Core/Cpu/Native/PosixHostStubs.cs +++ b/src/SharpEmu.Core/Cpu/Native/PosixHostStubs.cs @@ -322,20 +322,43 @@ internal static unsafe class PosixHostStubs private static nint EmitSleep(byte* page, ref int offset) { - // void Sleep(DWORD milliseconds in ecx) -> usleep(microseconds in edi). - var usleep = ResolveLibcExport("usleep"); + // void Sleep(DWORD milliseconds in ecx) -> nanosleep(&req, &rem), resumed on EINTR. usleep + // would be shorter, but macOS rejects usleep(>= 1 s) with EINVAL, and a signal (the + // runtime's thread activations, for one) cuts either short: a sleep that returned at once + // on an ignored error turned UE's idle HTTP thread into a hot spin. + var nanosleep = ResolveLibcExport("nanosleep"); + var errorLocation = ResolveLibcExport(OperatingSystem.IsMacOS() ? "__error" : "__errno_location"); var start = (nint)(page + offset); Emit(page, ref offset, 0x56); // push rsi Emit(page, ref offset, 0x57); // push rdi - Emit(page, ref offset, 0x48, 0x83, 0xEC, 0x08); // sub rsp, 8 - Emit(page, ref offset, 0x89, 0xCF); // mov edi, ecx - Emit(page, ref offset, 0x81, 0xFF, 0xFF, 0x0F, 0x00, 0x00); // cmp edi, 0xFFF - Emit(page, ref offset, 0x76, 0x05); // jbe +5 - Emit(page, ref offset, 0xBF, 0xFF, 0x0F, 0x00, 0x00); // mov edi, 0xFFF (cap at ~4s) - Emit(page, ref offset, 0x69, 0xFF, 0xE8, 0x03, 0x00, 0x00); // imul edi, edi, 1000 - EmitMovRaxImm64(page, ref offset, usleep); // mov rax, imm64 + Emit(page, ref offset, 0x53); // push rbx + Emit(page, ref offset, 0x48, 0x83, 0xEC, 0x20); // sub rsp, 0x20: req at [rsp], rem at [rsp+16] + Emit(page, ref offset, 0x89, 0xC8); // mov eax, ecx + Emit(page, ref offset, 0x31, 0xD2); // xor edx, edx + Emit(page, ref offset, 0xB9, 0xE8, 0x03, 0x00, 0x00); // mov ecx, 1000 + Emit(page, ref offset, 0xF7, 0xF1); // div ecx: eax = seconds, edx = milliseconds + Emit(page, ref offset, 0x48, 0x89, 0x04, 0x24); // mov [rsp], rax (tv_sec) + Emit(page, ref offset, 0x69, 0xD2, 0x40, 0x42, 0x0F, 0x00); // imul edx, edx, 1000000 + Emit(page, ref offset, 0x48, 0x89, 0x54, 0x24, 0x08); // mov [rsp+8], rdx (tv_nsec) + // loop: + Emit(page, ref offset, 0x48, 0x89, 0xE7); // mov rdi, rsp + Emit(page, ref offset, 0x48, 0x8D, 0x74, 0x24, 0x10); // lea rsi, [rsp+16] + EmitMovRaxImm64(page, ref offset, nanosleep); // mov rax, imm64 Emit(page, ref offset, 0xFF, 0xD0); // call rax - Emit(page, ref offset, 0x48, 0x83, 0xC4, 0x08); // add rsp, 8 + Emit(page, ref offset, 0x85, 0xC0); // test eax, eax + Emit(page, ref offset, 0x74, 0x26); // jz done + EmitMovRaxImm64(page, ref offset, errorLocation); // mov rax, imm64 + Emit(page, ref offset, 0xFF, 0xD0); // call rax + Emit(page, ref offset, 0x83, 0x38, 0x04); // cmp dword [rax], EINTR + Emit(page, ref offset, 0x75, 0x15); // jne done + Emit(page, ref offset, 0x48, 0x8B, 0x44, 0x24, 0x10); // mov rax, [rsp+16] + Emit(page, ref offset, 0x48, 0x89, 0x04, 0x24); // mov [rsp], rax + Emit(page, ref offset, 0x48, 0x8B, 0x44, 0x24, 0x18); // mov rax, [rsp+24] + Emit(page, ref offset, 0x48, 0x89, 0x44, 0x24, 0x08); // mov [rsp+8], rax + Emit(page, ref offset, 0xEB, 0xC2); // jmp loop + // done: + Emit(page, ref offset, 0x48, 0x83, 0xC4, 0x20); // add rsp, 0x20 + Emit(page, ref offset, 0x5B); // pop rbx Emit(page, ref offset, 0x5F); // pop rdi Emit(page, ref offset, 0x5E); // pop rsi Emit(page, ref offset, 0xC3); // ret diff --git a/src/SharpEmu.Core/Loader/GuestRedZonePatcher.cs b/src/SharpEmu.Core/Loader/GuestRedZonePatcher.cs index e03e727..33abb9d 100644 --- a/src/SharpEmu.Core/Loader/GuestRedZonePatcher.cs +++ b/src/SharpEmu.Core/Loader/GuestRedZonePatcher.cs @@ -20,7 +20,8 @@ internal static class GuestRedZonePatcher private const ulong AllocationAlignment = 0x10000; private const ulong MaximumRelativeJumpDistance = 0x7FFF_FFFF; private const int MaximumJumpTableEntries = 4096; - private const int MaximumJumpTableSetupInstructions = 6; + private const int MaximumJumpTableChainInstructions = 8; + private const int MaximumJumpTableRounds = 4; private const ulong LargeExecutableBytes = 32UL << 20; private static readonly TimeSpan ProgressInterval = TimeSpan.FromSeconds(2); @@ -425,202 +426,476 @@ internal static class GuestRedZonePatcher ulong functionEnd, HashSet branchTargets) { - List? resolvedTargets = null; - HashSet? instructionBoundaries = null; - for (var instructionIndex = 0; instructionIndex < decoded.Count; instructionIndex++) - { - if (decoded[instructionIndex].Instruction.FlowControl != FlowControl.IndirectBranch) - { - continue; - } - - instructionBoundaries ??= decoded - .Select(static entry => entry.Instruction.IP) - .ToHashSet(); - resolvedTargets ??= new List(); - if (!TryResolveJumpTable( - memory, - programHeaders, - imageBase, - decoded, - instructionIndex, - functionStart, - functionEnd, - instructionBoundaries, - branchTargets, - resolvedTargets)) - { - return false; - } - } - - if (resolvedTargets is not null) - { - branchTargets.UnionWith(resolvedTargets); - } - - return true; + return TryCollectJumpTableTargets( + (address, entries) => + IsRangeInReadableLoad(programHeaders, imageBase, address, (ulong)entries.Length) && + memory.TryRead(address, entries), + decoded, + functionStart, + functionEnd, + branchTargets); } - private static bool TryResolveJumpTable( - IVirtualMemory memory, - IReadOnlyList programHeaders, - ulong imageBase, + // Resolve every indirect jump of the function as a bounded relative jump table, or refuse. + // Bounds and table addresses are proven by walking the control-flow graph backwards + // from the table load. The graph includes the edges of the tables themselves, so the + // resolution is repeated until a round reproduces the targets it was built from. + private static bool TryCollectJumpTableTargets( + JumpTableReader readTable, IReadOnlyList decoded, + ulong functionStart, + ulong functionEnd, + HashSet branchTargets) + { + List? jumps = null; + for (var instructionIndex = 0; instructionIndex < decoded.Count; instructionIndex++) + { + if (decoded[instructionIndex].Instruction.FlowControl == FlowControl.IndirectBranch) + { + (jumps ??= []).Add(instructionIndex); + } + } + + if (jumps is null) + { + return true; + } + + var indexByAddress = new Dictionary(decoded.Count); + for (var instructionIndex = 0; instructionIndex < decoded.Count; instructionIndex++) + { + indexByAddress[decoded[instructionIndex].Instruction.IP] = instructionIndex; + } + + var infoFactory = new InstructionInfoFactory(); + var tableTargets = new Dictionary(); + for (var round = 0; round < MaximumJumpTableRounds; round++) + { + var predecessors = BuildPredecessors(decoded, indexByAddress, tableTargets); + var resolved = new Dictionary(); + foreach (var jumpIndex in jumps) + { + if (!TryResolveJumpTable( + readTable, + infoFactory, + decoded, + predecessors, + indexByAddress, + jumpIndex, + functionStart, + functionEnd, + out var targets)) + { + return false; + } + + resolved[jumpIndex] = targets; + } + + if (resolved.Count == tableTargets.Count && + resolved.All(entry => tableTargets.TryGetValue(entry.Key, out var previous) && + previous.AsSpan().SequenceEqual(entry.Value))) + { + foreach (var targets in resolved.Values) + { + branchTargets.UnionWith(targets); + } + + return true; + } + + tableTargets = resolved; + } + + return false; + } + + private static List<(int Index, bool Fallthrough)>?[] BuildPredecessors( + IReadOnlyList decoded, + Dictionary indexByAddress, + Dictionary tableTargets) + { + var predecessors = new List<(int Index, bool Fallthrough)>?[decoded.Count]; + for (var index = 0; index < decoded.Count; index++) + { + var instruction = decoded[index].Instruction; + switch (instruction.FlowControl) + { + case FlowControl.UnconditionalBranch: + case FlowControl.IndirectBranch: + case FlowControl.Return: + case FlowControl.Exception: + break; + default: + if (index + 1 < decoded.Count && decoded[index + 1].Instruction.IP == instruction.NextIP) + { + (predecessors[index + 1] ??= []).Add((index, true)); + } + + break; + } + + if (instruction.FlowControl is FlowControl.UnconditionalBranch or FlowControl.ConditionalBranch && + TryGetNearBranchTarget(instruction, out var target) && + indexByAddress.TryGetValue(target, out var targetIndex)) + { + (predecessors[targetIndex] ??= []).Add((index, false)); + } + + if (instruction.FlowControl == FlowControl.IndirectBranch && + tableTargets.TryGetValue(index, out var targets)) + { + foreach (var tableTarget in targets) + { + (predecessors[indexByAddress[tableTarget]] ??= []).Add((index, false)); + } + } + } + + return predecessors; + } + + private static bool HasOnlyFallthroughPredecessor( + List<(int Index, bool Fallthrough)>?[] predecessors, + int index) + { + return predecessors[index] is [(var previous, true)] && previous == index - 1; + } + + // jmp T, preceded by add T, B and movsxd T, dword [B + I*4] in one straight-line run. + private static bool TryResolveJumpTable( + JumpTableReader readTable, + InstructionInfoFactory infoFactory, + IReadOnlyList decoded, + List<(int Index, bool Fallthrough)>?[] predecessors, + Dictionary indexByAddress, int jumpIndex, ulong functionStart, ulong functionEnd, - IReadOnlySet instructionBoundaries, - IReadOnlySet knownBranchTargets, - List targets) + out ulong[] targets) { - if (jumpIndex < 4) - { - return false; - } - + targets = []; var jump = decoded[jumpIndex].Instruction; - var add = decoded[jumpIndex - 1].Instruction; - var load = decoded[jumpIndex - 2].Instruction; if (jump.OpCount != 1 || jump.GetOpKind(0) != OpKind.Register || - add.Mnemonic != Mnemonic.Add || add.OpCount != 2 || - add.GetOpKind(0) != OpKind.Register || add.GetOpKind(1) != OpKind.Register || - load.Mnemonic != Mnemonic.Movsxd || load.OpCount != 2 || - load.GetOpKind(0) != OpKind.Register || load.GetOpKind(1) != OpKind.Memory || - load.MemorySize.GetSize() != sizeof(int) || load.MemoryIndexScale != sizeof(int) || - load.MemoryBase == Register.None || load.MemoryIndex == Register.None || - load.MemoryDisplacement64 != 0) + jump.GetOpRegister(0).GetSize() != sizeof(ulong)) { return false; } - var targetRegister = jump.GetOpRegister(0).GetFullRegister(); - var tableRegister = add.GetOpRegister(1).GetFullRegister(); - var selectorRegister = load.MemoryIndex.GetFullRegister(); - if (targetRegister != add.GetOpRegister(0).GetFullRegister() || - targetRegister != load.GetOpRegister(0).GetFullRegister() || - tableRegister != load.MemoryBase.GetFullRegister() || - targetRegister.GetSize() != sizeof(ulong) || - tableRegister.GetSize() != sizeof(ulong) || - selectorRegister.GetSize() != sizeof(ulong) || - tableRegister == selectorRegister) - { - return false; - } - - var branchIndex = -1; - for (var index = jumpIndex - 3; - index >= 1 && index >= jumpIndex - MaximumJumpTableSetupInstructions; + var targetRegister = jump.GetOpRegister(0); + var tableRegister = Register.None; + var loadIndex = -1; + for (var index = jumpIndex - 1; + index >= 0 && index >= jumpIndex - MaximumJumpTableChainInstructions; index--) { + if (!HasOnlyFallthroughPredecessor(predecessors, index + 1)) + { + return false; + } + var candidate = decoded[index].Instruction; - if (candidate.Mnemonic == Mnemonic.Ja) + if (tableRegister == Register.None) { - branchIndex = index; - break; + if (!WritesRegister(infoFactory, candidate, targetRegister)) + { + continue; + } + + if (candidate.Mnemonic != Mnemonic.Add || candidate.OpCount != 2 || + candidate.GetOpKind(0) != OpKind.Register || candidate.GetOpKind(1) != OpKind.Register || + candidate.GetOpRegister(0) != targetRegister || + candidate.GetOpRegister(1).GetSize() != sizeof(ulong) || + candidate.GetOpRegister(1) == targetRegister) + { + return false; + } + + tableRegister = candidate.GetOpRegister(1); + continue; } - if (candidate.Mnemonic != Mnemonic.Lea) - { - return false; - } - } - - if (branchIndex < 1) - { - return false; - } - - for (var index = branchIndex + 1; index <= jumpIndex; index++) - { - if (knownBranchTargets.Contains(decoded[index].Instruction.IP)) - { - return false; - } - } - - var boundsBranch = decoded[branchIndex].Instruction; - var boundsCompare = decoded[branchIndex - 1].Instruction; - if (boundsBranch.FlowControl != FlowControl.ConditionalBranch || - !TryGetNearBranchTarget(boundsBranch, out var defaultTarget) || - defaultTarget < functionStart || defaultTarget >= functionEnd || - boundsCompare.Mnemonic != Mnemonic.Cmp || boundsCompare.OpCount != 2 || - boundsCompare.GetOpKind(0) != OpKind.Register || - boundsCompare.GetOpRegister(0).GetFullRegister() != selectorRegister || - boundsCompare.GetOpRegister(0).GetSize() != sizeof(int) || - !TryGetNonNegativeImmediate(boundsCompare, 1, out var maximumIndex) || - maximumIndex >= MaximumJumpTableEntries) - { - return false; - } - - var tableAddress = 0UL; - var tableDefinitionFound = false; - for (var index = branchIndex + 1; index <= jumpIndex - 3; index++) - { - var candidate = decoded[index].Instruction; - if (candidate.OpCount < 1 || candidate.GetOpKind(0) != OpKind.Register) + if (WritesRegister(infoFactory, candidate, tableRegister)) { return false; } - var writtenRegister = candidate.GetOpRegister(0).GetFullRegister(); - if (writtenRegister == selectorRegister) - { - return false; - } - - if (writtenRegister != tableRegister) + if (!WritesRegister(infoFactory, candidate, targetRegister)) { continue; } - if (tableDefinitionFound || candidate.OpCount != 2 || - candidate.GetOpKind(1) != OpKind.Memory || !candidate.IsIPRelativeMemoryOperand) + if (candidate.Mnemonic != Mnemonic.Movsxd || candidate.OpCount != 2 || + candidate.GetOpKind(0) != OpKind.Register || candidate.GetOpKind(1) != OpKind.Memory || + candidate.GetOpRegister(0) != targetRegister || + candidate.MemorySize.GetSize() != sizeof(int) || candidate.MemoryIndexScale != sizeof(int) || + candidate.MemoryBase != tableRegister || candidate.MemoryDisplacement64 != 0 || + candidate.MemoryIndex.GetSize() != sizeof(ulong) || + candidate.MemoryIndex == tableRegister || candidate.SegmentPrefix != Register.None) { return false; } - tableAddress = candidate.IPRelativeMemoryAddress; - tableDefinitionFound = true; + loadIndex = index; + break; } - if (!tableDefinitionFound) + if (loadIndex < 0) + { + return false; + } + + var selectorRegister = decoded[loadIndex].Instruction.MemoryIndex; + if (!TryProveSelectorBound(infoFactory, decoded, predecessors, loadIndex, selectorRegister, out var maximumIndex) || + !TryProveTableAddress(infoFactory, decoded, predecessors, loadIndex, tableRegister, out var tableAddress)) { return false; } var entryCount = checked((int)maximumIndex + 1); - var tableBytes = checked(entryCount * sizeof(int)); - if (!IsRangeInReadableLoad(programHeaders, imageBase, tableAddress, (ulong)tableBytes)) + var entries = GC.AllocateUninitializedArray(checked(entryCount * sizeof(int))); + if (!readTable(tableAddress, entries)) { return false; } - var entries = GC.AllocateUninitializedArray(tableBytes); - if (!memory.TryRead(tableAddress, entries)) - { - return false; - } - - var firstTarget = targets.Count; + var resolved = new ulong[entryCount]; for (var index = 0; index < entryCount; index++) { var relative = BinaryPrimitives.ReadInt32LittleEndian(entries.AsSpan(index * sizeof(int))); if (!TryAddSignedOffset(tableAddress, relative, out var target) || target < functionStart || target >= functionEnd || - !instructionBoundaries.Contains(target)) + !indexByAddress.ContainsKey(target)) { - targets.RemoveRange(firstTarget, targets.Count - firstTarget); return false; } - targets.Add(target); + resolved[index] = target; } + targets = resolved; return true; } + // Every path into the load must leave the fallthrough of ja after an unsigned compare of + // the selector, then reach the load through register copies and zero-extensions only. + // A compare narrower than the selector bounds it only when a zero-extension on that path + // cleared the bits above the compared ones. + private static bool TryProveSelectorBound( + InstructionInfoFactory infoFactory, + IReadOnlyList decoded, + List<(int Index, bool Fallthrough)>?[] predecessors, + int loadIndex, + Register selectorRegister, + out ulong maximumIndex) + { + maximumIndex = 0; + var bounded = false; + var visited = new HashSet<(int Index, Register Register, int Bits)>(); + var pending = new Stack<(int Index, Register Register, int Bits)>(); + pending.Push((loadIndex, selectorRegister, 64)); + while (pending.TryPop(out var state)) + { + if (state.Index == 0) + { + return false; + } + + if (predecessors[state.Index] is not { } incoming) + { + continue; + } + + foreach (var (predecessorIndex, fallthrough) in incoming) + { + var predecessor = decoded[predecessorIndex].Instruction; + if (fallthrough && predecessor.Mnemonic == Mnemonic.Ja && + TryGetNearBranchTarget(predecessor, out var branchTarget) && + branchTarget != predecessor.NextIP && + predecessorIndex >= 1 && + HasOnlyFallthroughPredecessor(predecessors, predecessorIndex) && + TryGetSelectorCompareBound(decoded[predecessorIndex - 1].Instruction, state.Register, state.Bits, out var bound)) + { + maximumIndex = Math.Max(maximumIndex, bound); + bounded = true; + continue; + } + + var register = state.Register; + var bits = state.Bits; + if (WritesRegister(infoFactory, predecessor, register) && + !TryTraceSelectorDefinition(predecessor, ref register, ref bits)) + { + return false; + } + + if (visited.Add((predecessorIndex, register, bits))) + { + pending.Push((predecessorIndex, register, bits)); + } + } + } + + return bounded; + } + + private static bool TryGetSelectorCompareBound(in Instruction compare, Register register, int bits, out ulong bound) + { + bound = 0; + if (compare.Mnemonic != Mnemonic.Cmp || compare.OpCount != 2 || + compare.GetOpKind(0) != OpKind.Register || + compare.GetOpRegister(0) is Register.AH or Register.CH or Register.DH or Register.BH || + compare.GetOpRegister(0).GetFullRegister() != register.GetFullRegister() || + !TryGetNonNegativeImmediate(compare, 1, out bound) || + bound >= MaximumJumpTableEntries) + { + return false; + } + + // A 32-bit compare of a 64-bit selector keeps the long-standing assumption that + // compilers emit it only for a zero-extended 32-bit value. + var compareBits = compare.GetOpRegister(0).GetSize() * 8; + return compareBits >= bits || compareBits == 32; + } + + // Follow the selector through movzx and register moves; any other definition is unbounded. + private static bool TryTraceSelectorDefinition(in Instruction definition, ref Register register, ref int bits) + { + if (definition.OpCount != 2 || + definition.GetOpKind(0) != OpKind.Register || definition.GetOpKind(1) != OpKind.Register || + definition.GetOpRegister(0).GetFullRegister() != register.GetFullRegister()) + { + return false; + } + + var destinationSize = definition.GetOpRegister(0).GetSize(); + var source = definition.GetOpRegister(1); + var sourceSize = source.GetSize(); + if (!source.IsGPR() || destinationSize < sizeof(int) || + source is Register.AH or Register.CH or Register.DH or Register.BH) + { + return false; + } + + switch (definition.Mnemonic) + { + case Mnemonic.Movzx when sourceSize < sizeof(int): + case Mnemonic.Mov when sourceSize == destinationSize: + register = source; + bits = Math.Min(bits, sourceSize * 8); + return true; + default: + return false; + } + } + + // Every path into the load must define the table register with lea of the same RIP-relative table. + private static bool TryProveTableAddress( + InstructionInfoFactory infoFactory, + IReadOnlyList decoded, + List<(int Index, bool Fallthrough)>?[] predecessors, + int loadIndex, + Register tableRegister, + out ulong tableAddress) + { + tableAddress = 0; + var defined = false; + var visited = new HashSet(); + var pending = new Stack(); + pending.Push(loadIndex); + while (pending.TryPop(out var index)) + { + if (index == 0) + { + return false; + } + + if (predecessors[index] is not { } incoming) + { + continue; + } + + foreach (var (predecessorIndex, _) in incoming) + { + var predecessor = decoded[predecessorIndex].Instruction; + if (!WritesRegister(infoFactory, predecessor, tableRegister)) + { + if (visited.Add(predecessorIndex)) + { + pending.Push(predecessorIndex); + } + + continue; + } + + if (predecessor.Mnemonic != Mnemonic.Lea || predecessor.OpCount != 2 || + predecessor.GetOpKind(0) != OpKind.Register || + predecessor.GetOpRegister(0) != tableRegister || + !predecessor.IsIPRelativeMemoryOperand || + (defined && predecessor.IPRelativeMemoryAddress != tableAddress)) + { + return false; + } + + tableAddress = predecessor.IPRelativeMemoryAddress; + defined = true; + } + } + + return defined; + } + + private static bool WritesRegister(InstructionInfoFactory infoFactory, in Instruction instruction, Register register) + { + var fullRegister = register.GetFullRegister(); + if (instruction.FlowControl is FlowControl.Call or FlowControl.IndirectCall && + fullRegister is Register.RAX or Register.RCX or Register.RDX or Register.RSI or Register.RDI or + Register.R8 or Register.R9 or Register.R10 or Register.R11) + { + return true; + } + + foreach (var used in infoFactory.GetInfo(instruction).GetUsedRegisters()) + { + if (used.Access is OpAccess.Write or OpAccess.CondWrite or OpAccess.ReadWrite or OpAccess.ReadCondWrite && + used.Register.GetFullRegister() == fullRegister) + { + return true; + } + } + + return false; + } + + // Test entry point: the code occupies the first codeLength bytes, the tables follow it. + internal static bool TryResolveJumpTableTargets( + byte[] image, + ulong baseAddress, + int codeLength, + out ulong[] targets) + { + var functionEnd = baseAddress + (ulong)codeLength; + var decoded = DecodeFunction(image, baseAddress, baseAddress, functionEnd); + var resolved = new HashSet(); + var success = TryCollectJumpTableTargets( + (address, entries) => + { + if (address < baseAddress || address - baseAddress + (ulong)entries.Length > (ulong)image.Length) + { + return false; + } + + image.AsSpan((int)(address - baseAddress), entries.Length).CopyTo(entries); + return true; + }, + decoded, + baseAddress, + functionEnd, + resolved); + targets = [.. resolved.Order()]; + return success; + } + private static bool IsRangeInReadableLoad( IReadOnlyList programHeaders, ulong imageBase, @@ -1431,6 +1706,8 @@ internal static class GuestRedZonePatcher private static ulong AlignDown(ulong value, ulong alignment) => value & ~(alignment - 1); + private delegate bool JumpTableReader(ulong address, byte[] entries); + private readonly record struct DecodedInstruction(Instruction Instruction); /// diff --git a/src/SharpEmu.Core/Memory/PhysicalVirtualMemory.cs b/src/SharpEmu.Core/Memory/PhysicalVirtualMemory.cs index fe3b026..b0e0854 100644 --- a/src/SharpEmu.Core/Memory/PhysicalVirtualMemory.cs +++ b/src/SharpEmu.Core/Memory/PhysicalVirtualMemory.cs @@ -140,12 +140,13 @@ public sealed unsafe class PhysicalVirtualMemory : IVirtualMemory, IGuestMemoryA private GuestSpaceOwner? _backedSpace; public PhysicalVirtualMemory(IHostMemory? hostMemory = null, IHostViewMemory? viewHost = null, - ulong backingBytes = GuestMemoryLayout.BackingBytes, bool preReserveGuestAddressSpace = false) + ulong backingBytes = 0, bool preReserveGuestAddressSpace = false) { _hostMemory = hostMemory ?? CrossPlatformHostMemory.Instance; if (viewHost != null) { - _backedSpace = new GuestSpaceOwner(viewHost, backingBytes, preReserveGuestAddressSpace); + // Zero means the whole guest backing (direct plus flexible memory). + _backedSpace = new GuestSpaceOwner(viewHost, backingBytes == 0 ? GuestMemoryLayout.BackingBytes : backingBytes, preReserveGuestAddressSpace); RunBackingSelfTest(); } } diff --git a/src/SharpEmu.Core/Runtime/SharpEmuRuntime.cs b/src/SharpEmu.Core/Runtime/SharpEmuRuntime.cs index f6d0c67..fddbf84 100644 --- a/src/SharpEmu.Core/Runtime/SharpEmuRuntime.cs +++ b/src/SharpEmu.Core/Runtime/SharpEmuRuntime.cs @@ -1401,6 +1401,15 @@ public sealed class SharpEmuRuntime : ISharpEmuRuntime { Console.Error.WriteLine("[LOADER][INFO] " + gpuSummary); } + // The presenter shuts down on its own thread; its caches still hold page watches + // and may still run queued GPU work until it detaches its stores. + if (!_gpuMemory.WaitForStoresDetached(TimeSpan.FromSeconds(15))) + { + Console.Error.WriteLine( + "[RUNTIME] The GPU caches did not release guest memory in time; keeping it mapped."); + return; + } + GuestGpuMemoryHook.Attach(null); _gpuMemory.Dispose(); } diff --git a/src/SharpEmu.GUI/GuiSettings.cs b/src/SharpEmu.GUI/GuiSettings.cs index 1e48bba..af2068f 100644 --- a/src/SharpEmu.GUI/GuiSettings.cs +++ b/src/SharpEmu.GUI/GuiSettings.cs @@ -70,6 +70,28 @@ public sealed class GuiSettings public string Resolution { get; set; } = "1920x1080"; + public string GuestResolution { get; set; } = "1920x1080"; + + internal static string NormalizeGuestResolution(string? value) => + string.Equals(value, "3840x2160", StringComparison.OrdinalIgnoreCase) ? "3840x2160" : "1920x1080"; + + // Host resolution multiplier of the render targets the renderer is allowed to scale. + public float RenderScale { get; set; } = 1.0f; + + internal static float NormalizeRenderScale(float value) => + SharpEmu.Libs.VideoOut.HostVideoOptions.NormalizeRenderScale(value); + + internal static readonly string[] RenderScaleChoices = ["50%", "75%", "100%", "125%", "150%", "200%"]; + + internal static string RenderScaleText(float value) => + ((int)MathF.Round(NormalizeRenderScale(value) * 100)).ToString(System.Globalization.CultureInfo.InvariantCulture) + "%"; + + internal static float ParseRenderScale(string? text) => + int.TryParse(text?.TrimEnd('%'), System.Globalization.NumberStyles.Integer, + System.Globalization.CultureInfo.InvariantCulture, out var percent) + ? NormalizeRenderScale(percent / 100.0f) + : 1.0f; + public int DisplayIndex { get; set; } public int RefreshRate { get; set; } @@ -89,9 +111,6 @@ public sealed class GuiSettings public string BinkPlaybackMode { get; set; } = "Guest"; - /// Internal render resolution scale (1.0 = native, 0.5 = half). - public double RenderResolutionScale { get; set; } = 1.0; - /// /// Discord application ID used for Rich Presence; the default is the /// SharpEmu application. Override to rebrand what Discord shows as @@ -184,6 +203,8 @@ public sealed class GuiSettings "Ukrainian"); settings.WindowMode = NormalizeChoice(settings.WindowMode, "Windowed", "Borderless", "Exclusive"); settings.Resolution = NormalizeResolution(settings.Resolution); + settings.GuestResolution = NormalizeGuestResolution(settings.GuestResolution); + settings.RenderScale = NormalizeRenderScale(settings.RenderScale); settings.ScalingMode = NormalizeChoice(settings.ScalingMode, "Fit", "Cover", "Stretch", "Integer"); settings.HdrMode = NormalizeChoice(settings.HdrMode, "Auto", "On", "Off"); settings.BinkPlaybackMode = NormalizeChoice(settings.BinkPlaybackMode, "Guest", "Host", "Skip"); diff --git a/src/SharpEmu.GUI/Languages/ar.json b/src/SharpEmu.GUI/Languages/ar.json index 7c7c318..7484ea7 100644 --- a/src/SharpEmu.GUI/Languages/ar.json +++ b/src/SharpEmu.GUI/Languages/ar.json @@ -219,6 +219,10 @@ "Options.WindowMode.Exclusive": "حصري", "Options.Resolution.Label": "الدقة", "Options.Resolution.Desc": "حجم النافذة الأولي أو دقة ملء الشاشة الحصرية.", + "Options.GuestResolution.Label": "وضع عرض اللعبة", + "Options.GuestResolution.Desc": "دقة العرض المُبلّغ عنها للعبة. الافتراضي 1080p. تختار الألعاب دقة التصيير الداخلية؛ وقد تتجاهل الألعاب ذات الدقة الثابتة هذا الإعداد.", + "Options.RenderScale.Label": "مقياس الدقة الداخلية", + "Options.RenderScale.Desc": "مضاعف دقة المضيف المطبّق على أهداف التصيير بحجم الشاشة في اللعبة. أقل من 100% يقايض جودة الصورة بالسرعة، وأكثر من 100% يفعل العكس. تبقى خرائط الظلال وجداول البحث والمواد المضغوطة والأسطح التي تكتبها مظللات الحوسبة بحجمها الأصلي دائمًا.", "Options.Display.Label": "الشاشة", "Options.Display.Desc": "الشاشة المستخدمة للتوسيط وملء الشاشة.", "Options.RefreshRate.Label": "معدل التحديث", diff --git a/src/SharpEmu.GUI/Languages/az.json b/src/SharpEmu.GUI/Languages/az.json index 8a418d1..00900ff 100644 --- a/src/SharpEmu.GUI/Languages/az.json +++ b/src/SharpEmu.GUI/Languages/az.json @@ -74,6 +74,10 @@ "Options.WindowMode.Exclusive": "Eksklüziv", "Options.Resolution.Label": "Təsvir ölçüsü", "Options.Resolution.Desc": "Pəncərənin başlanğıc ölçüsü və ya eksklüziv tam ekranın təsvir ölçüsü.", + "Options.GuestResolution.Label": "Oyunun ekran rejimi", + "Options.GuestResolution.Desc": "Oyuna bildirilən ekran təsvir ölçüsü. Standart 1080p-dir. Daxili render ölçüsünü oyun seçir; sabit ölçülü oyunlar bu ayarı nəzərə almaya bilər.", + "Options.RenderScale.Label": "Daxili təsvir ölçüsü əmsalı", + "Options.RenderScale.Desc": "Oyunun ekran ölçüsündəki render hədəflərinə tətbiq olunan host təsvir ölçüsü vuruğu. 100%-dən aşağı təsvir keyfiyyətini sürətə dəyişir, yuxarı isə əksinə. Kölgə xəritələri, axtarış cədvəlləri, sıxılmış teksturlar və hesablama şeyderlərinin yazdığı səthlər həmişə öz ölçüsündə qalır.", "Options.Display.Label": "Ekran", "Options.Display.Desc": "Pəncərənin mərkəzə yerləşdirilməsi və tam ekran üçün istifadə olunan monitor.", "Options.RefreshRate.Label": "Yenilənmə tezliyi", diff --git a/src/SharpEmu.GUI/Languages/br.json b/src/SharpEmu.GUI/Languages/br.json index 964ebbf..6c20521 100644 --- a/src/SharpEmu.GUI/Languages/br.json +++ b/src/SharpEmu.GUI/Languages/br.json @@ -220,6 +220,10 @@ "Options.WindowMode.Exclusive": "Exclusiva", "Options.Resolution.Label": "Resolução", "Options.Resolution.Desc": "Tamanho inicial da janela ou resolução de tela cheia exclusiva.", + "Options.GuestResolution.Label": "Modo de exibição do jogo", + "Options.GuestResolution.Desc": "Resolução informada ao jogo. O padrão é 1080p. O jogo escolhe a resolução interna de renderização; jogos com resolução fixa podem ignorar esta opção.", + "Options.RenderScale.Label": "Escala de resolução interna", + "Options.RenderScale.Desc": "Multiplicador da resolução do host aplicado aos alvos de renderização do tamanho da tela. Abaixo de 100% troca qualidade de imagem por velocidade; acima de 100% faz o contrário. Mapas de sombra, tabelas de consulta, texturas comprimidas e superfícies escritas por shaders de computação sempre mantêm o tamanho nativo.", "Options.Display.Label": "Tela", "Options.Display.Desc": "Monitor usado para centralização e tela cheia.", "Options.RefreshRate.Label": "Taxa de atualização", diff --git a/src/SharpEmu.GUI/Languages/de.json b/src/SharpEmu.GUI/Languages/de.json index e4b6b76..e9e16aa 100644 --- a/src/SharpEmu.GUI/Languages/de.json +++ b/src/SharpEmu.GUI/Languages/de.json @@ -219,6 +219,10 @@ "Options.WindowMode.Exclusive": "Exklusiv", "Options.Resolution.Label": "Auflösung", "Options.Resolution.Desc": "Anfängliche Fenstergröße oder exklusive Vollbildauflösung.", + "Options.GuestResolution.Label": "Anzeigemodus des Spiels", + "Options.GuestResolution.Desc": "Dem Spiel gemeldete Bildschirmauflösung. Standard ist 1080p. Spiele wählen ihre interne Renderauflösung selbst; Spiele mit fester Auflösung können diese Einstellung ignorieren.", + "Options.RenderScale.Label": "Interne Auflösungsskalierung", + "Options.RenderScale.Desc": "Host-Auflösungsfaktor für die bildschirmgroßen Renderziele des Spiels. Unter 100 % wird Bildqualität gegen Geschwindigkeit getauscht, über 100 % umgekehrt. Shadow Maps, Lookup-Tabellen, komprimierte Texturen und von Compute-Shadern beschriebene Oberflächen behalten immer ihre native Größe.", "Options.Display.Label": "Anzeige", "Options.Display.Desc": "Monitor für Zentrierung und Vollbilddarstellung.", "Options.RefreshRate.Label": "Bildwiederholrate", diff --git a/src/SharpEmu.GUI/Languages/dk.json b/src/SharpEmu.GUI/Languages/dk.json index b10d989..b36393f 100644 --- a/src/SharpEmu.GUI/Languages/dk.json +++ b/src/SharpEmu.GUI/Languages/dk.json @@ -219,6 +219,10 @@ "Options.WindowMode.Exclusive": "Eksklusiv", "Options.Resolution.Label": "Opløsning", "Options.Resolution.Desc": "Oprindelig vinduesstørrelse eller opløsning i eksklusiv fuldskærm.", + "Options.GuestResolution.Label": "Spillets skærmtilstand", + "Options.GuestResolution.Desc": "Skærmopløsning oplyst til spillet. Standard er 1080p. Spillet vælger den interne renderopløsning; spil med fast opløsning kan ignorere denne indstilling.", + "Options.RenderScale.Label": "Intern opløsningsskalering", + "Options.RenderScale.Desc": "Værtsopløsningsfaktor for spillets skærmstore rendermål. Under 100 % byttes billedkvalitet for hastighed; over 100 % omvendt. Skyggekort, opslagstabeller, komprimerede teksturer og overflader skrevet af compute-shaders beholder altid deres oprindelige størrelse.", "Options.Display.Label": "Skærm", "Options.Display.Desc": "Skærm, der bruges til centrering og fuldskærm.", "Options.RefreshRate.Label": "Opdateringshastighed", diff --git a/src/SharpEmu.GUI/Languages/ee.json b/src/SharpEmu.GUI/Languages/ee.json index c6cef57..fda62d0 100644 --- a/src/SharpEmu.GUI/Languages/ee.json +++ b/src/SharpEmu.GUI/Languages/ee.json @@ -85,6 +85,10 @@ "Options.WindowMode.Exclusive": "Eksklusiivne", "Options.Resolution.Label": "Eraldusvõime", "Options.Resolution.Desc": "Algne akna suurus või eksklusiivse täisekraani eraldusvõime.", + "Options.GuestResolution.Label": "Mängu kuvarežiim", + "Options.GuestResolution.Desc": "Mängule teatatud ekraani eraldusvõime. Vaikimisi 1080p. Mäng valib sisemise renderduse eraldusvõime; fikseeritud eraldusvõimega mängud võivad seda sätet eirata.", + "Options.RenderScale.Label": "Sisemise eraldusvõime skaala", + "Options.RenderScale.Desc": "Hosti eraldusvõime kordaja mängu ekraanisuuruste renderdussihtide jaoks. Alla 100% vahetab pildikvaliteedi kiiruse vastu, üle 100% vastupidi. Varjukaardid, otsingutabelid, pakitud tekstuurid ja arvutusvarjundajate kirjutatud pinnad jäävad alati oma algsesse suurusesse.", "Options.Display.Label": "Kuvar", "Options.Display.Desc": "Monitor, mida kasutatakse tsentreerimiseks ja täisekraanrežiimiks.", "Options.RefreshRate.Label": "Värskendussagedus", diff --git a/src/SharpEmu.GUI/Languages/en.json b/src/SharpEmu.GUI/Languages/en.json index cedc742..4740c3a 100644 --- a/src/SharpEmu.GUI/Languages/en.json +++ b/src/SharpEmu.GUI/Languages/en.json @@ -85,6 +85,10 @@ "Options.WindowMode.Exclusive": "Exclusive", "Options.Resolution.Label": "Resolution", "Options.Resolution.Desc": "Initial window size or exclusive fullscreen resolution.", + "Options.GuestResolution.Label": "Game display mode", + "Options.GuestResolution.Desc": "Display resolution reported to the game. Defaults to 1080p. Games choose their internal rendering resolution; fixed-resolution games may ignore this setting.", + "Options.RenderScale.Label": "Internal resolution scale", + "Options.RenderScale.Desc": "Host resolution multiplier applied to the game's screen-sized render targets. Below 100% trades image quality for speed; above 100% does the opposite. Shadow maps, lookup tables, compressed textures and compute-written surfaces always stay at their native size.", "Options.Display.Label": "Display", "Options.Display.Desc": "Monitor used for centering and fullscreen.", "Options.RefreshRate.Label": "Refresh rate", diff --git a/src/SharpEmu.GUI/Languages/es.json b/src/SharpEmu.GUI/Languages/es.json index ed8abdb..20752ce 100644 --- a/src/SharpEmu.GUI/Languages/es.json +++ b/src/SharpEmu.GUI/Languages/es.json @@ -220,6 +220,10 @@ "Options.WindowMode.Exclusive": "Exclusiva", "Options.Resolution.Label": "Resolución", "Options.Resolution.Desc": "Tamaño inicial de la ventana o resolución de pantalla completa exclusiva.", + "Options.GuestResolution.Label": "Modo de pantalla del juego", + "Options.GuestResolution.Desc": "Resolución de pantalla comunicada al juego. El valor predeterminado es 1080p. El juego elige su resolución interna de renderizado; los juegos con resolución fija pueden ignorar este ajuste.", + "Options.RenderScale.Label": "Escala de resolución interna", + "Options.RenderScale.Desc": "Multiplicador de resolución del anfitrión aplicado a los destinos de renderizado del tamaño de la pantalla. Por debajo del 100 % se cambia calidad de imagen por velocidad; por encima, lo contrario. Los mapas de sombras, las tablas de consulta, las texturas comprimidas y las superficies escritas por sombreadores de cómputo conservan siempre su tamaño nativo.", "Options.Display.Label": "Pantalla", "Options.Display.Desc": "Monitor utilizado para centrar y mostrar a pantalla completa.", "Options.RefreshRate.Label": "Frecuencia de actualización", diff --git a/src/SharpEmu.GUI/Languages/fr.json b/src/SharpEmu.GUI/Languages/fr.json index 0c425e2..7beaef1 100644 --- a/src/SharpEmu.GUI/Languages/fr.json +++ b/src/SharpEmu.GUI/Languages/fr.json @@ -220,6 +220,10 @@ "Options.WindowMode.Exclusive": "Exclusif", "Options.Resolution.Label": "Résolution", "Options.Resolution.Desc": "Taille initiale de la fenêtre ou résolution du plein écran exclusif.", + "Options.GuestResolution.Label": "Mode d’affichage du jeu", + "Options.GuestResolution.Desc": "Résolution d’écran indiquée au jeu. La valeur par défaut est 1080p. Le jeu choisit sa résolution de rendu interne ; les jeux à résolution fixe peuvent ignorer ce réglage.", + "Options.RenderScale.Label": "Échelle de résolution interne", + "Options.RenderScale.Desc": "Multiplicateur de résolution hôte appliqué aux cibles de rendu plein écran du jeu. En dessous de 100 %, la qualité d’image est échangée contre de la vitesse ; au-dessus, l’inverse. Les cartes d’ombres, les tables de correspondance, les textures compressées et les surfaces écrites par les shaders de calcul gardent toujours leur taille native.", "Options.Display.Label": "Écran", "Options.Display.Desc": "Moniteur utilisé pour le centrage et le plein écran.", "Options.RefreshRate.Label": "Fréquence de rafraîchissement", diff --git a/src/SharpEmu.GUI/Languages/hu.json b/src/SharpEmu.GUI/Languages/hu.json index af24a81..68d45a5 100644 --- a/src/SharpEmu.GUI/Languages/hu.json +++ b/src/SharpEmu.GUI/Languages/hu.json @@ -220,6 +220,10 @@ "Options.WindowMode.Exclusive": "Kizárólagos", "Options.Resolution.Label": "Felbontás", "Options.Resolution.Desc": "Kezdeti ablakméret vagy kizárólagos teljes képernyős felbontás.", + "Options.GuestResolution.Label": "A játék megjelenítési módja", + "Options.GuestResolution.Desc": "A játéknak jelzett képernyőfelbontás. Alapértelmezés: 1080p. A belső renderelési felbontást a játék választja; a rögzített felbontású játékok figyelmen kívül hagyhatják ezt a beállítást.", + "Options.RenderScale.Label": "Belső felbontás skálázása", + "Options.RenderScale.Desc": "A gazdagép felbontásszorzója a játék képernyőméretű rendercéljaira. 100% alatt a képminőséget cseréli sebességre, 100% fölött fordítva. Az árnyéktérképek, a keresőtáblák, a tömörített textúrák és a compute shaderek által írt felületek mindig natív méretükben maradnak.", "Options.Display.Label": "Kijelző", "Options.Display.Desc": "A középre igazításhoz és teljes képernyőhöz használt monitor.", "Options.RefreshRate.Label": "Frissítési gyakoriság", diff --git a/src/SharpEmu.GUI/Languages/it.json b/src/SharpEmu.GUI/Languages/it.json index 97a6885..0822364 100644 --- a/src/SharpEmu.GUI/Languages/it.json +++ b/src/SharpEmu.GUI/Languages/it.json @@ -224,6 +224,10 @@ "Options.WindowMode.Exclusive": "Esclusiva", "Options.Resolution.Label": "Risoluzione", "Options.Resolution.Desc": "Dimensione iniziale della finestra o risoluzione dello schermo intero esclusivo.", + "Options.GuestResolution.Label": "Modalità schermo del gioco", + "Options.GuestResolution.Desc": "Risoluzione dello schermo comunicata al gioco. Il valore predefinito è 1080p. Il gioco sceglie la risoluzione interna di rendering; i giochi con risoluzione fissa possono ignorare questa impostazione.", + "Options.RenderScale.Label": "Scala della risoluzione interna", + "Options.RenderScale.Desc": "Moltiplicatore della risoluzione host applicato ai target di rendering a schermo intero del gioco. Sotto il 100% si scambia qualità d'immagine con velocità; sopra il 100% avviene il contrario. Shadow map, tabelle di consultazione, texture compresse e superfici scritte dai compute shader restano sempre alla dimensione nativa.", "Options.Display.Label": "Schermo", "Options.Display.Desc": "Monitor utilizzato per il centraggio e lo schermo intero.", "Options.RefreshRate.Label": "Frequenza di aggiornamento", diff --git a/src/SharpEmu.GUI/Languages/ja.json b/src/SharpEmu.GUI/Languages/ja.json index 697da9d..56429b8 100644 --- a/src/SharpEmu.GUI/Languages/ja.json +++ b/src/SharpEmu.GUI/Languages/ja.json @@ -219,6 +219,10 @@ "Options.WindowMode.Exclusive": "排他", "Options.Resolution.Label": "解像度", "Options.Resolution.Desc": "初期ウィンドウサイズまたは排他フルスクリーンの解像度。", + "Options.GuestResolution.Label": "ゲームの表示モード", + "Options.GuestResolution.Desc": "ゲームに通知する画面解像度。既定値は1080pです。内部描画解像度はゲームが決定するため、解像度が固定されたゲームではこの設定が反映されない場合があります。", + "Options.RenderScale.Label": "内部解像度スケール", + "Options.RenderScale.Desc": "ゲームの画面サイズのレンダーターゲットに適用するホスト解像度の倍率です。100%未満は画質と引き換えに速度を得て、100%超はその逆になります。シャドウマップ、ルックアップテーブル、圧縮テクスチャ、コンピュートシェーダーが書き込むサーフェスは常に元のサイズのままです。", "Options.Display.Label": "ディスプレイ", "Options.Display.Desc": "中央配置とフルスクリーンに使用するモニター。", "Options.RefreshRate.Label": "リフレッシュレート", diff --git a/src/SharpEmu.GUI/Languages/ko.json b/src/SharpEmu.GUI/Languages/ko.json index 443677a..a49aee2 100644 --- a/src/SharpEmu.GUI/Languages/ko.json +++ b/src/SharpEmu.GUI/Languages/ko.json @@ -219,6 +219,10 @@ "Options.WindowMode.Exclusive": "독점", "Options.Resolution.Label": "해상도", "Options.Resolution.Desc": "초기 창 크기 또는 독점 전체 화면 해상도.", + "Options.GuestResolution.Label": "게임 디스플레이 모드", + "Options.GuestResolution.Desc": "게임에 전달하는 화면 해상도입니다. 기본값은 1080p입니다. 내부 렌더링 해상도는 게임이 결정하며, 해상도가 고정된 게임은 이 설정을 무시할 수 있습니다.", + "Options.RenderScale.Label": "내부 해상도 배율", + "Options.RenderScale.Desc": "게임의 화면 크기 렌더 타깃에 적용되는 호스트 해상도 배율입니다. 100% 미만은 화질을 속도와 맞바꾸고 100% 초과는 그 반대입니다. 섀도 맵, 룩업 테이블, 압축 텍스처, 컴퓨트 셰이더가 쓰는 표면은 항상 원래 크기를 유지합니다.", "Options.Display.Label": "디스플레이", "Options.Display.Desc": "가운데 배치와 전체 화면에 사용할 모니터.", "Options.RefreshRate.Label": "새로 고침 빈도", diff --git a/src/SharpEmu.GUI/Languages/nl.json b/src/SharpEmu.GUI/Languages/nl.json index 0c83373..a133e84 100644 --- a/src/SharpEmu.GUI/Languages/nl.json +++ b/src/SharpEmu.GUI/Languages/nl.json @@ -219,6 +219,10 @@ "Options.WindowMode.Exclusive": "Exclusief", "Options.Resolution.Label": "Resolutie", "Options.Resolution.Desc": "Initiële venstergrootte of resolutie voor exclusief volledig scherm.", + "Options.GuestResolution.Label": "Weergavemodus van het spel", + "Options.GuestResolution.Desc": "Schermresolutie die aan het spel wordt doorgegeven. Standaard 1080p. Het spel kiest de interne renderresolutie; spellen met een vaste resolutie kunnen deze instelling negeren.", + "Options.RenderScale.Label": "Interne resolutieschaal", + "Options.RenderScale.Desc": "Hostresolutievermenigvuldiger voor de schermgrote rendertargets van het spel. Onder 100% ruil je beeldkwaliteit in voor snelheid; boven 100% andersom. Schaduwkaarten, opzoektabellen, gecomprimeerde texturen en door compute-shaders beschreven oppervlakken behouden altijd hun oorspronkelijke grootte.", "Options.Display.Label": "Beeldscherm", "Options.Display.Desc": "Monitor die wordt gebruikt voor centrering en volledig scherm.", "Options.RefreshRate.Label": "Verversingssnelheid", diff --git a/src/SharpEmu.GUI/Languages/pl.json b/src/SharpEmu.GUI/Languages/pl.json index f15bb64..75f882f 100644 --- a/src/SharpEmu.GUI/Languages/pl.json +++ b/src/SharpEmu.GUI/Languages/pl.json @@ -85,6 +85,10 @@ "Options.WindowMode.Exclusive": "Wyłączne", "Options.Resolution.Label": "Rozdzielczość", "Options.Resolution.Desc": "Początkowy rozmiar okna lub rozdzielczość pełnego ekranu wyłącznego.", + "Options.GuestResolution.Label": "Tryb wyświetlania gry", + "Options.GuestResolution.Desc": "Rozdzielczość ekranu zgłaszana grze. Domyślnie 1080p. Gra wybiera wewnętrzną rozdzielczość renderowania; gry ze stałą rozdzielczością mogą zignorować to ustawienie.", + "Options.RenderScale.Label": "Skala rozdzielczości wewnętrznej", + "Options.RenderScale.Desc": "Mnożnik rozdzielczości hosta dla pełnoekranowych buforów renderowania gry. Poniżej 100% oddaje jakość obrazu za wydajność; powyżej 100% odwrotnie. Mapy cieni, tablice LUT, tekstury skompresowane i powierzchnie zapisywane przez shadery obliczeniowe zawsze zachowują natywny rozmiar.", "Options.Display.Label": "Wyświetlacz", "Options.Display.Desc": "Monitor używany do centrowania i trybu pełnoekranowego.", "Options.RefreshRate.Label": "Częstotliwość odświeżania", diff --git a/src/SharpEmu.GUI/Languages/pt.json b/src/SharpEmu.GUI/Languages/pt.json index 27f1d15..bca186e 100644 --- a/src/SharpEmu.GUI/Languages/pt.json +++ b/src/SharpEmu.GUI/Languages/pt.json @@ -220,6 +220,10 @@ "Options.WindowMode.Exclusive": "Exclusivo", "Options.Resolution.Label": "Resolução", "Options.Resolution.Desc": "Tamanho inicial da janela ou resolução de ecrã inteiro exclusivo.", + "Options.GuestResolution.Label": "Modo de apresentação do jogo", + "Options.GuestResolution.Desc": "Resolução do ecrã comunicada ao jogo. A predefinição é 1080p. O jogo escolhe a resolução interna de renderização; jogos com resolução fixa podem ignorar esta opção.", + "Options.RenderScale.Label": "Escala de resolução interna", + "Options.RenderScale.Desc": "Multiplicador da resolução do anfitrião aplicado aos alvos de renderização do tamanho do ecrã. Abaixo de 100% troca qualidade de imagem por velocidade; acima de 100% faz o contrário. Mapas de sombras, tabelas de consulta, texturas comprimidas e superfícies escritas por sombreadores de computação mantêm sempre o tamanho nativo.", "Options.Display.Label": "Ecrã", "Options.Display.Desc": "Monitor utilizado para centrar e apresentar em ecrã inteiro.", "Options.RefreshRate.Label": "Taxa de atualização", diff --git a/src/SharpEmu.GUI/Languages/ru.json b/src/SharpEmu.GUI/Languages/ru.json index 7bbd4e8..af1dfa1 100644 --- a/src/SharpEmu.GUI/Languages/ru.json +++ b/src/SharpEmu.GUI/Languages/ru.json @@ -81,6 +81,10 @@ "Options.WindowMode.Exclusive": "Эксклюзивный", "Options.Resolution.Label": "Разрешение", "Options.Resolution.Desc": "Начальный размер окна или разрешение эксклюзивного полноэкранного режима.", + "Options.GuestResolution.Label": "Режим экрана игры", + "Options.GuestResolution.Desc": "Разрешение экрана, сообщаемое игре. По умолчанию 1080p. Игра выбирает внутреннее разрешение рендеринга; игры с фиксированным разрешением могут игнорировать эту настройку.", + "Options.RenderScale.Label": "Масштаб внутреннего разрешения", + "Options.RenderScale.Desc": "Множитель разрешения хоста для полноэкранных целей рендеринга игры. Ниже 100% качество изображения обменивается на скорость, выше 100% — наоборот. Карты теней, таблицы поиска, сжатые текстуры и поверхности, записываемые вычислительными шейдерами, всегда сохраняют исходный размер.", "Options.Display.Label": "Монитор", "Options.Display.Desc": "Монитор, используемый для центрирования окна и полноэкранного режима.", "Options.RefreshRate.Label": "Частота обновления", diff --git a/src/SharpEmu.GUI/Languages/tr.json b/src/SharpEmu.GUI/Languages/tr.json index 6135c15..e3cab4e 100644 --- a/src/SharpEmu.GUI/Languages/tr.json +++ b/src/SharpEmu.GUI/Languages/tr.json @@ -63,6 +63,10 @@ "Options.WindowMode.Desc": "Normal pencere, kenarlıksız masaüstü veya özel tam ekran.", "Options.Resolution.Label": "Çözünürlük", "Options.Resolution.Desc": "Başlangıç pencere boyutu veya özel tam ekran çözünürlüğü.", + "Options.GuestResolution.Label": "Oyunun ekran modu", + "Options.GuestResolution.Desc": "Oyuna bildirilen ekran çözünürlüğü. Varsayılan 1080p’dir. Dahili işleme çözünürlüğünü oyun seçer; sabit çözünürlüklü oyunlar bu ayarı yok sayabilir.", + "Options.RenderScale.Label": "Dahili çözünürlük ölçeği", + "Options.RenderScale.Desc": "Oyunun ekran boyutundaki işleme hedeflerine uygulanan ana makine çözünürlük çarpanı. %100'ün altında görüntü kalitesi hıza takas edilir, üzerinde ise tersi olur. Gölge haritaları, arama tabloları, sıkıştırılmış dokular ve hesaplama gölgelendiricilerinin yazdığı yüzeyler her zaman kendi boyutlarında kalır.", "Options.Display.Label": "Ekran", "Options.Display.Desc": "Ortalama ve tam ekran için kullanılan monitör.", "Options.RefreshRate.Label": "Yenileme hızı", diff --git a/src/SharpEmu.GUI/MainWindow.GameOptions.cs b/src/SharpEmu.GUI/MainWindow.GameOptions.cs index 2f42710..d6fdae4 100644 --- a/src/SharpEmu.GUI/MainWindow.GameOptions.cs +++ b/src/SharpEmu.GUI/MainWindow.GameOptions.cs @@ -45,6 +45,8 @@ public partial class MainWindow GameLogLevelBox.ItemsSource = _logLevelChoices; GameWindowModeBox.ItemsSource = _windowModeChoices; GameScalingModeBox.ItemsSource = _scalingModeChoices; + GameGuestResolutionBox.ItemsSource = new[] { "1920x1080", "3840x2160" }; + GameRenderScaleBox.ItemsSource = GuiSettings.RenderScaleChoices; GameHdrModeBox.ItemsSource = _hdrModeChoices; GameOverlayModeBox.ItemsSource = _overlayModeChoices; GameOverlayCornerBox.ItemsSource = _overlayCornerChoices; @@ -115,6 +117,8 @@ public partial class MainWindow GameRefreshRateBox.SelectionChanged += (_, _) => PersistOpenGameSettings(); GameScalingModeBox.SelectionChanged += (_, _) => PersistOpenGameSettings(); GameVSyncToggle.IsCheckedChanged += (_, _) => PersistOpenGameSettings(); + GameGuestResolutionBox.SelectionChanged += (_, _) => PersistOpenGameSettings(); + GameRenderScaleBox.SelectionChanged += (_, _) => PersistOpenGameSettings(); GameHdrModeBox.SelectionChanged += (_, _) => PersistOpenGameSettings(); GameOverlayEnabledToggle.IsCheckedChanged += (_, _) => PersistOpenGameSettings(); GameOverlayModeBox.SelectionChanged += (_, _) => PersistOpenGameSettings(); @@ -216,6 +220,8 @@ public partial class MainWindow _scalingModeChoices, effective.ScalingMode, "Fit"); + GameGuestResolutionBox.SelectedItem = effective.GuestResolution; + GameRenderScaleBox.SelectedItem = GuiSettings.RenderScaleText(effective.RenderScale); GameVSyncToggle.IsChecked = effective.VSync; GameOverlayEnabledToggle.IsChecked = effective.OverlayEnabled; GameOverlayModeBox.SelectedItem = FindChoice(_overlayModeChoices, effective.OverlayMode, "TitleBar"); @@ -277,6 +283,8 @@ public partial class MainWindow LogToFile = GameLogToFileToggle.IsChecked == true, WindowMode = SelectedComboText(GameWindowModeBox, "Windowed"), Resolution = SelectedComboText(GameResolutionBox, "1920x1080"), + GuestResolution = SelectedComboText(GameGuestResolutionBox, "1920x1080"), + RenderScale = GuiSettings.ParseRenderScale(SelectedComboText(GameRenderScaleBox, "100%")), DisplayIndex = GameDisplayBox.SelectedItem is HostDisplayOption display ? display.Index : 0, diff --git a/src/SharpEmu.GUI/MainWindow.axaml b/src/SharpEmu.GUI/MainWindow.axaml index 2c601fb..4d04979 100644 --- a/src/SharpEmu.GUI/MainWindow.axaml +++ b/src/SharpEmu.GUI/MainWindow.axaml @@ -871,6 +871,22 @@ SPDX-License-Identifier: GPL-2.0-or-later VerticalAlignment="Center" /> + + + + + + + + + + + + _settings.WindowMode = SelectedComboText(WindowModeBox, "Windowed"); DisplayBox.SelectionChanged += (_, _) => OnHostDisplayChanged(); ResolutionBox.SelectionChanged += (_, _) => OnHostResolutionChanged(); + GuestResolutionBox.SelectionChanged += (_, _) => _settings.GuestResolution = SelectedComboText(GuestResolutionBox, "1920x1080"); + RenderScaleBox.SelectionChanged += (_, _) => + _settings.RenderScale = GuiSettings.ParseRenderScale(SelectedComboText(RenderScaleBox, "100%")); RefreshRateBox.SelectionChanged += (_, _) => OnHostRefreshRateChanged(); ScalingModeBox.SelectionChanged += (_, _) => _settings.ScalingMode = SelectedComboText(ScalingModeBox, "Fit"); VSyncToggle.IsCheckedChanged += (_, _) => _settings.VSync = VSyncToggle.IsChecked == true; @@ -1373,6 +1376,8 @@ public partial class MainWindow : Window LogLevelBox.ItemsSource = _logLevelChoices; WindowModeBox.ItemsSource = _windowModeChoices; ScalingModeBox.ItemsSource = _scalingModeChoices; + GuestResolutionBox.ItemsSource = new[] { "1920x1080", "3840x2160" }; + RenderScaleBox.ItemsSource = GuiSettings.RenderScaleChoices; HdrModeBox.ItemsSource = _hdrModeChoices; BinkPlaybackBox.ItemsSource = _binkPlaybackChoices; OverlayModeBox.ItemsSource = _overlayModeChoices; @@ -1448,6 +1453,8 @@ public partial class MainWindow : Window LoadHostDisplayOptions(); ScalingModeBox.SelectedIndex = ChoiceIndex(_settings.ScalingMode, "Fit", "Cover", "Stretch", "Integer"); VSyncToggle.IsChecked = _settings.VSync; + GuestResolutionBox.SelectedItem = GuiSettings.NormalizeGuestResolution(_settings.GuestResolution); + RenderScaleBox.SelectedItem = GuiSettings.RenderScaleText(_settings.RenderScale); HdrModeBox.SelectedIndex = ChoiceIndex(_settings.HdrMode, "Auto", "On", "Off"); OverlayEnabledToggle.IsChecked = _settings.OverlayEnabled; OverlayModeBox.SelectedIndex = ChoiceIndex(_settings.OverlayMode, "Full", "Minimal", "TitleBar"); @@ -2891,13 +2898,6 @@ public partial class MainWindow : Window }); _appliedEnvironmentVariables.Add(BinkModeEnvironmentName); - Environment.SetEnvironmentVariable( - "SHARPEMU_RENDER_SCALE", - _settings.RenderResolutionScale.ToString( - "0.###", - System.Globalization.CultureInfo.InvariantCulture)); - _appliedEnvironmentVariables.Add("SHARPEMU_RENDER_SCALE"); - foreach (var entry in customEnvironment) { if (!CustomEnvironmentVariables.TryParseEntry(entry, out var name, out var value)) @@ -3115,6 +3115,9 @@ public partial class MainWindow : Window arguments.Add($"--window-mode={launch.Settings.WindowMode.ToLowerInvariant()}"); arguments.Add($"--resolution={launch.Settings.Resolution}"); + arguments.Add($"--guest-resolution={launch.Settings.GuestResolution}"); + arguments.Add( + $"--render-scale={launch.Settings.RenderScale.ToString(System.Globalization.CultureInfo.InvariantCulture)}"); arguments.Add($"--display={launch.Settings.DisplayIndex}"); arguments.Add($"--refresh-rate={launch.Settings.RefreshRate}"); arguments.Add($"--scaling={launch.Settings.ScalingMode.ToLowerInvariant()}"); diff --git a/src/SharpEmu.GUI/PerGameSettings.cs b/src/SharpEmu.GUI/PerGameSettings.cs index 58b2cff..3a2aa40 100644 --- a/src/SharpEmu.GUI/PerGameSettings.cs +++ b/src/SharpEmu.GUI/PerGameSettings.cs @@ -26,6 +26,10 @@ public sealed class PerGameSettings public string? Resolution { get; set; } + public string? GuestResolution { get; set; } + + public float? RenderScale { get; set; } + public int? DisplayIndex { get; set; } public int? RefreshRate { get; set; } @@ -52,6 +56,8 @@ public sealed class PerGameSettings LogToFile is null && WindowMode is null && Resolution is null && + GuestResolution is null && + RenderScale is null && DisplayIndex is null && RefreshRate is null && ScalingMode is null && @@ -145,6 +151,17 @@ public sealed class PerGameSettings Resolution = null; } + if (string.Equals(GuestResolution, global.GuestResolution, StringComparison.OrdinalIgnoreCase)) + { + GuestResolution = null; + } + + if (RenderScale is { } renderScale && + GuiSettings.NormalizeRenderScale(renderScale) == GuiSettings.NormalizeRenderScale(global.RenderScale)) + { + RenderScale = null; + } + if (DisplayIndex == global.DisplayIndex) { DisplayIndex = null; @@ -272,6 +289,8 @@ public sealed record EffectiveLaunchSettings( bool LogToFile, string WindowMode, string Resolution, + string GuestResolution, + float RenderScale, int DisplayIndex, int RefreshRate, string ScalingMode, @@ -289,6 +308,8 @@ public sealed record EffectiveLaunchSettings( perGame?.LogToFile ?? global.LogToFile, perGame?.WindowMode ?? global.WindowMode, perGame?.Resolution ?? global.Resolution, + GuiSettings.NormalizeGuestResolution(perGame?.GuestResolution ?? global.GuestResolution), + GuiSettings.NormalizeRenderScale(perGame?.RenderScale ?? global.RenderScale), Math.Max(0, perGame?.DisplayIndex ?? global.DisplayIndex), Math.Clamp(perGame?.RefreshRate ?? global.RefreshRate, 0, 1000), perGame?.ScalingMode ?? global.ScalingMode, diff --git a/src/SharpEmu.HLE/GpuMemory/GuestGpuMemory.cs b/src/SharpEmu.HLE/GpuMemory/GuestGpuMemory.cs index a99f4b5..d7d7433 100644 --- a/src/SharpEmu.HLE/GpuMemory/GuestGpuMemory.cs +++ b/src/SharpEmu.HLE/GpuMemory/GuestGpuMemory.cs @@ -61,6 +61,23 @@ public sealed class GuestGpuMemory : IDisposable Volatile.Write(ref _images, images); } + // The host GPU detaches its stores after its caches have released every page watch. + public bool WaitForStoresDetached(TimeSpan timeout) + { + var deadline = Environment.TickCount64 + (long)timeout.TotalMilliseconds; + while (Buffers is not null || Images is not null) + { + if (Environment.TickCount64 >= deadline) + { + return false; + } + + Thread.Sleep(10); + } + + return true; + } + // Retry after recovery, including a watch removed before the fault reached its store. public bool TryResolveFault(FaultKind kind, ulong address) { @@ -294,8 +311,11 @@ public sealed class GuestGpuMemory : IDisposable } } - // Enter the GPU worker before the caller takes locks used by GPU memory reads. - public void RunMappingChange(Action change) + // Enter the GPU worker before the caller takes locks used by GPU memory reads. A change that + // only maps addresses no live mapping covers touches nothing the GPU can be using, so it + // skips the drain; removing or replacing a mapping must wait for the GPU first. needsDrain is + // evaluated where the change runs, after every earlier mapping change has been applied. + public void RunMappingChange(Action change, Func? needsDrain = null) { using var requestScope = GuestMemoryProfile.Measure(GuestMemoryProfile.Operation.MappingRequest); for (;;) @@ -322,7 +342,7 @@ public sealed class GuestGpuMemory : IDisposable void ApplyChange(IGpuTickScheduler? scheduler) { // Finish callbacks before the mapping transaction takes its locks. - if (scheduler is { Active: true }) + if (scheduler is { Active: true } && (needsDrain?.Invoke() ?? true)) { using var drainScope = GuestMemoryProfile.Measure(GuestMemoryProfile.Operation.MappingDrain); scheduler.FinishMemoryAccess(); diff --git a/src/SharpEmu.HLE/GpuMemory/GuestGpuMemoryHook.cs b/src/SharpEmu.HLE/GpuMemory/GuestGpuMemoryHook.cs index dedd136..b1eec63 100644 --- a/src/SharpEmu.HLE/GpuMemory/GuestGpuMemoryHook.cs +++ b/src/SharpEmu.HLE/GpuMemory/GuestGpuMemoryHook.cs @@ -8,8 +8,17 @@ public static class GuestGpuMemoryHook private static readonly ulong _tracePage = ParseTracePage(); private static readonly bool _traceFirstDeviceFault = string.Equals( Environment.GetEnvironmentVariable("SHARPEMU_TRACE_GPU_MEMORY_ADDRESS"), "auto", StringComparison.OrdinalIgnoreCase); + // "coherence": the page of the first GPU-copy/guest divergence (SHARPEMU_TRACE_COHERENCE) is traced from then on. + private static readonly bool _traceFirstDivergence = string.Equals( + Environment.GetEnvironmentVariable("SHARPEMU_TRACE_GPU_MEMORY_ADDRESS"), "coherence", StringComparison.OrdinalIgnoreCase); private static long _selectedTracePage; + public static void SelectDivergenceTracePage(ulong address) + { + if (!_traceFirstDivergence || !TrySelectTracePage(ref _selectedTracePage, address)) return; + Trace(address, 1, "divergence-trace-selected"); + } + public static void SelectDeviceFaultTracePage(ulong address) { if (!_traceFirstDeviceFault || !TrySelectTracePage(ref _selectedTracePage, address)) return; @@ -33,9 +42,9 @@ public static class GuestGpuMemoryHook } public static ulong TraceAddress => - _traceFirstDeviceFault ? unchecked((ulong)Interlocked.Read(ref _selectedTracePage)) : _tracePage; + _traceFirstDeviceFault || _traceFirstDivergence ? unchecked((ulong)Interlocked.Read(ref _selectedTracePage)) : _tracePage; - public static bool TraceEnabled => _traceFirstDeviceFault || _tracePage != 0; + public static bool TraceEnabled => _traceFirstDeviceFault || _traceFirstDivergence || _tracePage != 0; public static bool Traces(ulong address, ulong size) => OverlapsTracePage(TraceAddress, address, size); diff --git a/src/SharpEmu.HLE/GpuMemory/GuestPageTracker.cs b/src/SharpEmu.HLE/GpuMemory/GuestPageTracker.cs index 0ac893a..51f6f66 100644 --- a/src/SharpEmu.HLE/GpuMemory/GuestPageTracker.cs +++ b/src/SharpEmu.HLE/GpuMemory/GuestPageTracker.cs @@ -122,12 +122,40 @@ public sealed class GuestPageTracker return count; } - public void MarkCpuDirtyPages(ulong vaddr, ulong size) => Mark(vaddr, size, WriteOrigin.Cpu, enable: true, create: true); + // Reports whether at least one page transitioned from clean to CPU-dirty. + public bool MarkCpuDirtyPages(ulong vaddr, ulong size) => Mark(vaddr, size, WriteOrigin.Cpu, enable: true, create: true); public void MarkGpuDirtyPages(ulong vaddr, ulong size) => Mark(vaddr, size, WriteOrigin.Gpu, enable: true, create: true); public void ClearGpuDirtyPages(ulong vaddr, ulong size) => Mark(vaddr, size, WriteOrigin.Gpu, enable: false, create: false); + // Marks the tracked pages of a range GPU-dirty after the fact, page by page under the region + // lock. A page the CPU dirtied meanwhile is left CPU-owned and counted as a conflict, since the + // two cannot both own it. Returns the number of such conflicting pages. + public int MarkGpuWrittenPages(ulong vaddr, ulong size) + { + RejectUploadCallbackReentry(); + var conflicts = 0; + VisitRegions(vaddr, size, create: false, (region, offset, bytes) => + { + using var _ = region.Lock.Hold(); + for (var page = offset; page < offset + bytes; page += PageBytes) + { + var pageBytes = Math.Min(PageBytes, offset + bytes - page); + if (region.IsModified(WriteOrigin.Cpu, page, pageBytes)) + { + conflicts++; + continue; + } + + region.ChangeState(WriteOrigin.Gpu, enable: true, region.BaseAddress + page, pageBytes); + } + + return false; + }); + return conflicts; + } + // True when every page of the range has a region, so the range is under tracking. public bool HasRegion(ulong vaddr, ulong size) { @@ -143,6 +171,32 @@ public sealed class GuestPageTracker return true; } + // Hot pages stay writable and are copied whole at every sync, so a ring the CPU fills a + // little at a time would end up copied in full forever. Once a frame they lose their heat: + // the next sync copies them once more and protects them again, and only pages the CPU + // keeps rewriting within a frame turn hot again. + public void DecayCpuWriteHeat() + { + RejectUploadCallbackReentry(); + for (var index = 0; index < _regions.Length; index++) + { + if (Volatile.Read(ref _regions[index]) is not { HasCpuWriteHeat: true } region) + { + continue; + } + + region.Lock.Enter(); + try + { + region.ResetCpuWriteHeat(region.BaseAddress, TrackerLayout.BlockBytes); + } + finally + { + region.Lock.Exit(); + } + } + } + public void UntrackMemory(ulong vaddr, ulong size) { RejectUploadCallbackReentry(); @@ -242,50 +296,110 @@ public sealed class GuestPageTracker } } - // Region locks stay held across uploadFunc only for a written range, which turns GPU-dirty. + // Receives the CPU-dirty runs an upload copies, then performs the upload while the regions are held. + public interface IUploadRangeSink + { + void Range(ulong address, ulong size); + + void Upload(); + } + + private struct DelegateUploadSink(Action rangeFunc, Action uploadFunc) : IUploadRangeSink + { + public readonly void Range(ulong address, ulong size) => rangeFunc(address, size); + + public readonly void Upload() => uploadFunc(); + } + + // Forwards a region's upload runs to the sink and remembers the runs it cleared, for rollback. + private struct RegionUploadVisitor(TSink sink, List<(TrackedRegion Region, ulong Address, ulong Size)> cleared) + : TrackedRegion.ICpuUploadVisitor + where TSink : struct, IUploadRangeSink + { + public TSink Sink = sink; + + public readonly void Cleared(TrackedRegion region, ulong address, ulong size) => cleared.Add((region, address, size)); + + public void Upload(ulong address, ulong size) => Sink.Range(address, size); + } + + // Uploads run on every bound buffer of every draw, so the bookkeeping lists are reused per thread. + [ThreadStatic] + private static List? _scratchHeld; + + [ThreadStatic] + private static List<(TrackedRegion Region, ulong Address, ulong Size)>? _scratchCleared; + public void ForEachUploadRange(ulong vaddr, ulong size, bool isWritten, Action rangeFunc, Action uploadFunc, bool preserveCpuWriteHotPages = true) + { + var sink = new DelegateUploadSink(rangeFunc, uploadFunc); + ForEachUploadRange(vaddr, size, isWritten, ref sink, preserveCpuWriteHotPages); + } + + // Region locks stay held across the upload only for a written range, which turns GPU-dirty. + public void ForEachUploadRange(ulong vaddr, ulong size, bool isWritten, ref TSink sink, bool preserveCpuWriteHotPages = true, + bool skipCpuWriteHotPages = false) + where TSink : struct, IUploadRangeSink { RejectUploadCallbackReentry(); VisitRegions(vaddr, size, create: true, static (_, _, _) => false); var previousOwner = _uploadOwner; _uploadOwner = this; - var held = new List(); - var cleared = new List<(TrackedRegion Region, ulong Address, ulong Size)>(); + // Another tracker may upload from this callback. It needs its own rollback + // state; the reentry guard only rejects reentry into this tracker. + var held = previousOwner is null + ? _scratchHeld ??= new List() + : new List(); + var cleared = previousOwner is null + ? _scratchCleared ??= new List<(TrackedRegion Region, ulong Address, ulong Size)>() + : new List<(TrackedRegion Region, ulong Address, ulong Size)>(); + held.Clear(); + cleared.Clear(); try { using (GpuMemoryAccessProfile.Measure(GpuMemoryAccessProfile.Operation.UploadTracking, size)) { - VisitRegions(vaddr, size, create: false, (region, offset, bytes) => + var visitor = new RegionUploadVisitor(sink, cleared); + var preserveHotPages = !isWritten && preserveCpuWriteHotPages; + for (var (index, offset, remaining) = (vaddr / BlockBytes, vaddr % BlockBytes, size); remaining != 0; index++, offset = 0) { + var bytes = Math.Min(BlockBytes - offset, remaining); + remaining -= bytes; + if (Volatile.Read(ref _regions[index]) is not { } region) + { + continue; + } + region.Lock.Enter(); held.Add(region); - var address = region.BaseAddress + offset; - region.ForEachCpuUploadRange( - preserveHotPages: !isWritten && preserveCpuWriteHotPages, - address, - bytes, - (runAddress, runSize) => cleared.Add((region, runAddress, runSize)), - rangeFunc); + region.ForEachCpuUploadRange(preserveHotPages, region.BaseAddress + offset, bytes, ref visitor, + skipHotPages: !isWritten && skipCpuWriteHotPages); if (!isWritten) { region.Lock.Exit(); - held.Remove(region); + held.RemoveAt(held.Count - 1); } + } - return false; - }); + sink = visitor.Sink; } - uploadFunc(); + sink.Upload(); if (isWritten) { - VisitRegions(vaddr, size, create: false, (region, offset, bytes) => + for (var (index, offset, remaining) = (vaddr / BlockBytes, vaddr % BlockBytes, size); remaining != 0; index++, offset = 0) { + var bytes = Math.Min(BlockBytes - offset, remaining); + remaining -= bytes; + if (Volatile.Read(ref _regions[index]) is not { } region) + { + continue; + } + region.ChangeState(WriteOrigin.Gpu, enable: true, region.BaseAddress + offset, bytes); region.Lock.Exit(); held.Remove(region); - return false; - }); + } } } catch @@ -333,6 +447,8 @@ public sealed class GuestPageTracker region.Lock.Exit(); } + held.Clear(); + cleared.Clear(); _uploadOwner = previousOwner; } } @@ -368,22 +484,25 @@ public sealed class GuestPageTracker } } - private void Mark(ulong vaddr, ulong size, WriteOrigin side, bool enable, bool create) + private bool Mark(ulong vaddr, ulong size, WriteOrigin side, bool enable, bool create) { RejectUploadCallbackReentry(); + var changed = false; VisitRegions(vaddr, size, create, (region, offset, bytes) => { using var _ = region.Lock.Hold(); if (side == WriteOrigin.Cpu && enable) { - region.MarkCpuWrite(region.BaseAddress + offset, bytes); + changed |= region.MarkCpuWrite(region.BaseAddress + offset, bytes); } else { region.ChangeState(side, enable, region.BaseAddress + offset, bytes); + changed = true; } return false; }); + return changed; } private void VisitDirtyRanges(ulong vaddr, ulong size, WriteOrigin side, bool clear, Action visit) => diff --git a/src/SharpEmu.HLE/GpuMemory/TrackedRegion.cs b/src/SharpEmu.HLE/GpuMemory/TrackedRegion.cs index 7dfe95c..25e1add 100644 --- a/src/SharpEmu.HLE/GpuMemory/TrackedRegion.cs +++ b/src/SharpEmu.HLE/GpuMemory/TrackedRegion.cs @@ -72,9 +72,13 @@ public sealed class TrackedRegion } // Count only clean-to-dirty transitions. Repeated reads cannot make a page hot. - public void MarkCpuWrite(ulong address, ulong size) + // Returns whether this write changed ownership from clean to CPU-dirty. + // Callers use the result to avoid repeating protection bookkeeping for a + // page that was already CPU-owned. + public bool MarkCpuWrite(ulong address, ulong size) { var (start, end) = GetPageRange(address, size); + var changed = false; for (var page = start; page < end; page++) { if (_cpuDirty.Get(page)) @@ -82,6 +86,8 @@ public sealed class TrackedRegion continue; } + changed = true; + if (_recentCpuUploads.Get(page)) { if (_repeatedCpuWrites.Get(page)) @@ -101,7 +107,17 @@ public sealed class TrackedRegion } } + // The page is already writable and CPU-owned. Re-running ChangeState + // scans every page in the 4 MiB region and republishes the same summary, + // once for every managed guest-store; do neither when ownership did not + // change. Callers hold Lock while reading and updating these masks. + if (!changed) + { + return false; + } + ChangeState(WriteOrigin.Cpu, enable: true, address, size); + return true; } public void ChangeState(WriteOrigin side, bool enable, ulong address, ulong size) @@ -144,20 +160,32 @@ public sealed class TrackedRegion } } + // Receives the runs one upload clears and the runs it copies; a struct visitor keeps the walk allocation-free. + public interface ICpuUploadVisitor + { + void Cleared(TrackedRegion region, ulong address, ulong size); + + void Upload(ulong address, ulong size); + } + // Hot read-only pages stay writable and dirty, so each obtain observes current CPU bytes. - public void ForEachCpuUploadRange( - bool preserveHotPages, - ulong address, - ulong size, - Action visitCleared, - Action visitUpload) + // skipHotPages leaves them out entirely, for a caller that already copied them since the + // last memory visibility point. + public void ForEachCpuUploadRange(bool preserveHotPages, ulong address, ulong size, ref TVisitor visitor, + bool skipHotPages = false) + where TVisitor : struct, ICpuUploadVisitor { var (start, end) = GetPageRange(address, size); var upload = new PageMask(_cpuDirty, start, end); + if (skipHotPages) + { + upload &= ~_hotCpuWrites; + } + var cleared = preserveHotPages ? upload & ~_hotCpuWrites : upload; foreach (var (runStart, runEnd) in cleared) { - visitCleared(BaseAddress + (ulong)runStart * PageBytes, (ulong)(runEnd - runStart) * PageBytes); + visitor.Cleared(this, BaseAddress + (ulong)runStart * PageBytes, (ulong)(runEnd - runStart) * PageBytes); _cpuDirty.UnsetRange(runStart, runEnd); _recentCpuUploads.SetRange(runStart, runEnd); } @@ -165,10 +193,13 @@ public sealed class TrackedRegion UpdateCpuProtection(track: true); foreach (var (runStart, runEnd) in upload) { - visitUpload(BaseAddress + (ulong)runStart * PageBytes, (ulong)(runEnd - runStart) * PageBytes); + visitor.Upload(BaseAddress + (ulong)runStart * PageBytes, (ulong)(runEnd - runStart) * PageBytes); } } + // A racy hint for the per-frame decay; the decay itself runs under Lock. + public bool HasCpuWriteHeat => _hotCpuWrites.Any; + public void ResetCpuWriteHeat(ulong address, ulong size) { var (start, end) = GetPageRange(address, size); diff --git a/src/SharpEmu.HLE/GuestMemoryLayout.cs b/src/SharpEmu.HLE/GuestMemoryLayout.cs index a0204ae..a96f9f2 100644 --- a/src/SharpEmu.HLE/GuestMemoryLayout.cs +++ b/src/SharpEmu.HLE/GuestMemoryLayout.cs @@ -5,9 +5,16 @@ namespace SharpEmu.HLE; public static class GuestMemoryLayout { - public const ulong DirectBytes = 16384UL * 1024 * 1024; + // The direct memory a PS5 game gets: 12.5 GiB of the console's 16 GiB, the rest being the + // system's. Games size their pools from sceKernelGetDirectMemorySize, so reporting the + // whole 16 GiB made them claim (and fill) memory a console never gives them. + // SHARPEMU_DIRECT_MEMORY_MB overrides it. + public static readonly ulong DirectBytes = + (ulong.TryParse(Environment.GetEnvironmentVariable("SHARPEMU_DIRECT_MEMORY_MB"), out var megabytes) && megabytes > 0 + ? megabytes + : 12800UL) * 1024 * 1024; public const ulong FlexibleBytes = 448UL * 1024 * 1024; - public const ulong FlexibleOffset = DirectBytes; - public const ulong BackingBytes = DirectBytes + FlexibleBytes; + public static readonly ulong FlexibleOffset = DirectBytes; + public static readonly ulong BackingBytes = DirectBytes + FlexibleBytes; public const ulong GuestPage = 0x4000; } diff --git a/src/SharpEmu.HLE/Host/Posix/PosixHostStubs.cs b/src/SharpEmu.HLE/Host/Posix/PosixHostStubs.cs index 22ae8ec..c19ef0c 100644 --- a/src/SharpEmu.HLE/Host/Posix/PosixHostStubs.cs +++ b/src/SharpEmu.HLE/Host/Posix/PosixHostStubs.cs @@ -462,20 +462,43 @@ internal static unsafe class PosixHostStubs private static nint EmitSleep(byte* page, ref int offset) { - // void Sleep(DWORD milliseconds in ecx) -> usleep(microseconds in edi). - var usleep = ResolveLibcExport("usleep"); + // void Sleep(DWORD milliseconds in ecx) -> nanosleep(&req, &rem), resumed on EINTR. usleep + // would be shorter, but macOS rejects usleep(>= 1 s) with EINVAL, and a signal (the + // runtime's thread activations, for one) cuts either short: a sleep that returned at once + // on an ignored error turned UE's idle HTTP thread into a hot spin. + var nanosleep = ResolveLibcExport("nanosleep"); + var errorLocation = ResolveLibcExport(OperatingSystem.IsMacOS() ? "__error" : "__errno_location"); var start = (nint)(page + offset); Emit(page, ref offset, 0x56); // push rsi Emit(page, ref offset, 0x57); // push rdi - Emit(page, ref offset, 0x48, 0x83, 0xEC, 0x08); // sub rsp, 8 - Emit(page, ref offset, 0x89, 0xCF); // mov edi, ecx - Emit(page, ref offset, 0x81, 0xFF, 0xFF, 0x0F, 0x00, 0x00); // cmp edi, 0xFFF - Emit(page, ref offset, 0x76, 0x05); // jbe +5 - Emit(page, ref offset, 0xBF, 0xFF, 0x0F, 0x00, 0x00); // mov edi, 0xFFF (cap at ~4s) - Emit(page, ref offset, 0x69, 0xFF, 0xE8, 0x03, 0x00, 0x00); // imul edi, edi, 1000 - EmitMovRaxImm64(page, ref offset, usleep); // mov rax, imm64 + Emit(page, ref offset, 0x53); // push rbx + Emit(page, ref offset, 0x48, 0x83, 0xEC, 0x20); // sub rsp, 0x20: req at [rsp], rem at [rsp+16] + Emit(page, ref offset, 0x89, 0xC8); // mov eax, ecx + Emit(page, ref offset, 0x31, 0xD2); // xor edx, edx + Emit(page, ref offset, 0xB9, 0xE8, 0x03, 0x00, 0x00); // mov ecx, 1000 + Emit(page, ref offset, 0xF7, 0xF1); // div ecx: eax = seconds, edx = milliseconds + Emit(page, ref offset, 0x48, 0x89, 0x04, 0x24); // mov [rsp], rax (tv_sec) + Emit(page, ref offset, 0x69, 0xD2, 0x40, 0x42, 0x0F, 0x00); // imul edx, edx, 1000000 + Emit(page, ref offset, 0x48, 0x89, 0x54, 0x24, 0x08); // mov [rsp+8], rdx (tv_nsec) + // loop: + Emit(page, ref offset, 0x48, 0x89, 0xE7); // mov rdi, rsp + Emit(page, ref offset, 0x48, 0x8D, 0x74, 0x24, 0x10); // lea rsi, [rsp+16] + EmitMovRaxImm64(page, ref offset, nanosleep); // mov rax, imm64 Emit(page, ref offset, 0xFF, 0xD0); // call rax - Emit(page, ref offset, 0x48, 0x83, 0xC4, 0x08); // add rsp, 8 + Emit(page, ref offset, 0x85, 0xC0); // test eax, eax + Emit(page, ref offset, 0x74, 0x26); // jz done + EmitMovRaxImm64(page, ref offset, errorLocation); // mov rax, imm64 + Emit(page, ref offset, 0xFF, 0xD0); // call rax + Emit(page, ref offset, 0x83, 0x38, 0x04); // cmp dword [rax], EINTR + Emit(page, ref offset, 0x75, 0x15); // jne done + Emit(page, ref offset, 0x48, 0x8B, 0x44, 0x24, 0x10); // mov rax, [rsp+16] + Emit(page, ref offset, 0x48, 0x89, 0x04, 0x24); // mov [rsp], rax + Emit(page, ref offset, 0x48, 0x8B, 0x44, 0x24, 0x18); // mov rax, [rsp+24] + Emit(page, ref offset, 0x48, 0x89, 0x44, 0x24, 0x08); // mov [rsp+8], rax + Emit(page, ref offset, 0xEB, 0xC2); // jmp loop + // done: + Emit(page, ref offset, 0x48, 0x83, 0xC4, 0x20); // add rsp, 0x20 + Emit(page, ref offset, 0x5B); // pop rbx Emit(page, ref offset, 0x5F); // pop rdi Emit(page, ref offset, 0x5E); // pop rsi Emit(page, ref offset, 0xC3); // ret diff --git a/src/SharpEmu.HLE/Host/Posix/PosixViewRegions.cs b/src/SharpEmu.HLE/Host/Posix/PosixViewRegions.cs index 24869ab..ff5f998 100644 --- a/src/SharpEmu.HLE/Host/Posix/PosixViewRegions.cs +++ b/src/SharpEmu.HLE/Host/Posix/PosixViewRegions.cs @@ -1,6 +1,8 @@ // Copyright (C) 2026 SharpEmu Emulator Project // SPDX-License-Identifier: GPL-2.0-or-later +using System.Threading; + namespace SharpEmu.HLE.Host.Posix; // View mappings bypass the anonymous-allocation tables. Keep their query state together. @@ -11,6 +13,12 @@ internal static class PosixViewRegions uint[]? PageProtections, ulong PageBase); private static readonly List Regions = new(); + // The regions in address order, replaced whenever the list changes. Queries read it without + // Gate: every HLE import queries the guest stack's protection first, and Gate is held across + // mprotect by the GPU write tracker, so a locked query made each import wait on page traffic. + // Page protections change in place and are read racily, as a concurrent change always was. + private static Region[] _snapshot = []; + internal static void Replace(ulong address, ulong size, uint state, uint protection) { HostMemory.OnMappingChanged(); @@ -46,6 +54,8 @@ internal static class PosixViewRegions else Regions.Add(region); } + + Volatile.Write(ref _snapshot, [.. Regions]); } // Mapping creates the storage. Fault-time protection updates must not allocate. @@ -66,9 +76,8 @@ internal static class PosixViewRegions internal static bool TryQuery(ulong address, out HostMemory.BasicInfo info) { - lock (Gate) { - foreach (var region in Regions) + foreach (var region in Volatile.Read(ref _snapshot)) { if (region.Start > address) break; if (address >= region.End) continue; diff --git a/src/SharpEmu.HLE/HostMemory.cs b/src/SharpEmu.HLE/HostMemory.cs index 6586bf5..3e6dffa 100644 --- a/src/SharpEmu.HLE/HostMemory.cs +++ b/src/SharpEmu.HLE/HostMemory.cs @@ -149,6 +149,14 @@ public static unsafe class HostMemory private static readonly object Gate = new(); private static readonly SortedList Regions = new(); + // The regions in address order, replaced under Gate whenever a region is added or removed. + // Queries read it without the lock: Protect holds Gate across its mprotect call, and GPU + // write tracking protects pages constantly, so a locked query (one per HLE call reading + // stack arguments) waited tens of microseconds behind those syscalls. + private static Region[] RegionSnapshot = []; + + private static void PublishRegionsLocked() => Volatile.Write(ref RegionSnapshot, [.. Regions.Values]); + private sealed class Region { public ulong Base; @@ -156,6 +164,10 @@ public static unsafe class HostMemory public uint DefaultProtect; public Dictionary? PageProtects; + // The overridden pages in address order, so a query finds where a protection run ends + // without visiting every override: write tracking leaves tens of thousands of them. + public SortedSet? OverridePages; + public ulong End => Base + Size; public uint ProtectAt(ulong pageAddress) @@ -269,6 +281,7 @@ public static unsafe class HostMemory Size = alignedSize, DefaultProtect = protect }; + PublishRegionsLocked(); return (void*)result; } @@ -288,6 +301,7 @@ public static unsafe class HostMemory HostMemory.OnMappingChanged(); Regions.Remove((ulong)address); + PublishRegionsLocked(); return munmap((nint)address, (nuint)region.Size) == 0; } } @@ -325,6 +339,10 @@ public static unsafe class HostMemory { info = default; var pageAddress = AlignDown((ulong)address, PageSize); + if (TryQueryUnlocked(pageAddress, ref info)) + { + return (nuint)sizeof(BasicInfo); + } lock (Gate) { @@ -340,30 +358,37 @@ public static unsafe class HostMemory var pageProtects = region.PageProtects; uint protect; ulong runEnd; - if (pageProtects is null || pageProtects.Count == 0) + if (pageProtects is null || pageProtects.Count == 0 || region.OverridePages is not { } overridePages) { protect = region.DefaultProtect; runEnd = region.End; } else if (pageProtects.TryGetValue(pageAddress, out protect)) { + // Overrides run while the next page is overridden with the same protection. runEnd = pageAddress + PageSize; - while (runEnd < region.End && - pageProtects.TryGetValue(runEnd, out var nextProtect) && - nextProtect == protect) + foreach (var overrideAddress in runEnd < region.End ? overridePages.GetViewBetween(runEnd, region.End - 1) : []) { + if (overrideAddress != runEnd || pageProtects[overrideAddress] != protect) + { + break; + } + runEnd += PageSize; } } else { + // The default protection runs to the next override. protect = region.DefaultProtect; runEnd = region.End; - foreach (var overrideAddress in pageProtects.Keys) + if (pageAddress + PageSize < region.End) { - if (overrideAddress > pageAddress && overrideAddress < runEnd) + // The first element only: a view's Count walks the whole view. + foreach (var overrideAddress in overridePages.GetViewBetween(pageAddress + PageSize, region.End - 1)) { runEnd = overrideAddress; + break; } } } @@ -382,12 +407,18 @@ public static unsafe class HostMemory // reported as a free block reaching to the next tracked region // so scanning callers keep advancing. var nextBase = ulong.MaxValue; - foreach (var regionBase in Regions.Keys) + var keys = Regions.Keys; + for (int low = 0, high = keys.Count - 1; low <= high;) { - if (regionBase > pageAddress) + var middle = low + ((high - low) >> 1); + if (keys[middle] > pageAddress) { - nextBase = regionBase; - break; + nextBase = keys[middle]; + high = middle - 1; + } + else + { + low = middle + 1; } } @@ -416,6 +447,56 @@ public static unsafe class HostMemory return false; } + // Answers without Gate for untracked memory and for regions without per-page overrides; + // a region with overrides needs their dictionary, which only Gate protects. A concurrent + // Protect may land just before or after either way, as with the locked query. + private static bool TryQueryUnlocked(ulong pageAddress, ref BasicInfo info) + { + var regions = Volatile.Read(ref RegionSnapshot); + Region? candidate = null; + var nextBase = ulong.MaxValue; + for (int low = 0, high = regions.Length - 1; low <= high;) + { + var middle = low + ((high - low) >> 1); + if (regions[middle].Base <= pageAddress) + { + candidate = regions[middle]; + low = middle + 1; + } + else + { + nextBase = regions[middle].Base; + high = middle - 1; + } + } + + if (candidate is not null && pageAddress < candidate.End) + { + if (Volatile.Read(ref candidate.PageProtects) is { Count: > 0 }) + { + return false; + } + + info.BaseAddress = pageAddress; + info.AllocationBase = candidate.Base; + info.AllocationProtect = candidate.DefaultProtect; + info.RegionSize = candidate.End - pageAddress; + info.State = MEM_COMMIT; + info.Protect = candidate.DefaultProtect; + info.Type = MEM_PRIVATE; + return true; + } + + info.BaseAddress = pageAddress; + info.AllocationBase = 0; + info.AllocationProtect = PAGE_NOACCESS; + info.RegionSize = (nextBase == ulong.MaxValue ? pageAddress + PageSize : nextBase) - pageAddress; + info.State = MEM_FREE_STATE; + info.Protect = PAGE_NOACCESS; + info.Type = 0; + return true; + } + private static bool TryFindRegionLocked(ulong address, out Region region) { region = null!; @@ -454,20 +535,26 @@ public static unsafe class HostMemory { region.DefaultProtect = protect; region.PageProtects = null; + region.OverridePages = null; return; } region.PageProtects ??= new Dictionary(); + region.OverridePages ??= new SortedSet(); var end = start + size; for (var pageAddress = start; pageAddress < end; pageAddress += PageSize) { if (protect == region.DefaultProtect) { - region.PageProtects.Remove(pageAddress); + if (region.PageProtects.Remove(pageAddress)) + { + region.OverridePages.Remove(pageAddress); + } } else { region.PageProtects[pageAddress] = protect; + region.OverridePages.Add(pageAddress); } } } diff --git a/src/SharpEmu.LibAtrac9/Atrac9Decoder.cs b/src/SharpEmu.LibAtrac9/Atrac9Decoder.cs index fa76ff7..39cbd0c 100644 --- a/src/SharpEmu.LibAtrac9/Atrac9Decoder.cs +++ b/src/SharpEmu.LibAtrac9/Atrac9Decoder.cs @@ -48,6 +48,42 @@ namespace LibAtrac9 DecodeSuperFrame(pcmOut); } + /// + /// Decodes one frame from the beginning of + /// and returns the byte count consumed, including the frame's byte + /// alignment. The decoder state is retained so consecutive calls can + /// decode frames from the same superframe. + /// + public int DecodeFrame(byte[] atrac9Data, short[][] pcmOut, int outputOffset, int frameIndex) + { + if (!_initialized) throw new InvalidOperationException("Decoder must be initialized before decoding."); + if (atrac9Data == null) throw new ArgumentNullException(nameof(atrac9Data)); + if (pcmOut == null) throw new ArgumentNullException(nameof(pcmOut)); + if (pcmOut.Length < Config.ChannelCount || + outputOffset < 0 || + outputOffset > Config.FrameSamples * Config.FramesPerSuperframe - Config.FrameSamples || + frameIndex < 0 || + frameIndex >= Config.FramesPerSuperframe) + { + throw new ArgumentException("Invalid ATRAC9 frame decode buffers or index"); + } + + for (var i = 0; i < Config.ChannelCount; i++) + { + if (pcmOut[i] == null || pcmOut[i].Length < outputOffset + Config.FrameSamples) + { + throw new ArgumentException("PCM buffer is too small"); + } + } + + Reader.SetBuffer(atrac9Data); + Frame.FrameIndex = frameIndex; + DecodeFrame(Reader, Frame); + PcmFloatToShort(pcmOut, outputOffset); + Reader.AlignPosition(8); + return (Reader.Position + 7) / 8; + } + private void ValidateDecodeBuffers(byte[] atrac9Buffer, short[][] pcmBuffer) { if (atrac9Buffer == null) throw new ArgumentNullException(nameof(atrac9Buffer)); diff --git a/src/SharpEmu.LibAtrac9/Unpack.cs b/src/SharpEmu.LibAtrac9/Unpack.cs index 700869a..8cf1198 100644 --- a/src/SharpEmu.LibAtrac9/Unpack.cs +++ b/src/SharpEmu.LibAtrac9/Unpack.cs @@ -357,7 +357,14 @@ namespace LibAtrac9 int precision = channel.Precisions[i] + 1; if (precision <= maxHuffPrecision) { - HuffmanCodebook huff = Tables.HuffmanSpectrum[channel.CodebookSet[i]][precision][Tables.QuantUnitToCodebookIndex[i]]; + // A valid stream never selects a codebook the format leaves undefined. + HuffmanCodebook huff = Tables.HuffmanSpectrum[channel.CodebookSet[i]]?[precision]?[Tables.QuantUnitToCodebookIndex[i]]; + if (huff == null) + { + throw new InvalidDataException( + $"The ATRAC9 block selects an undefined spectrum codebook: set={channel.CodebookSet[i]} precision={precision} unit={i}."); + } + int groupCount = subbandCount >> huff.ValueCountPower; for (int j = 0; j < groupCount; j++) { diff --git a/src/SharpEmu.Libs/Acm/AcmExports.cs b/src/SharpEmu.Libs/Acm/AcmExports.cs index 813a9db..9330b4c 100644 --- a/src/SharpEmu.Libs/Acm/AcmExports.cs +++ b/src/SharpEmu.Libs/Acm/AcmExports.cs @@ -170,8 +170,51 @@ public static class AcmExports ExportName = "sceAcm_ConvReverb_SharedInput", Target = Generation.Gen4 | Generation.Gen5, LibraryName = "libSceAcm")] - public static int AcmConvReverbSharedInput(CpuContext ctx) => - AdvanceBatchInfo(ctx, 1024); + public static int AcmConvReverbSharedInput(CpuContext ctx) + { + var outputCount = unchecked((uint)ctx[CpuRegister.Rcx]); + var outputDescriptors = ReadStackArg64(ctx, 0); + if (outputDescriptors != 0) + { + Span descriptor = stackalloc byte[24]; + for (uint index = 0; index < Math.Min(outputCount, 64u); index++) + { + if (!ctx.TryReadUInt64(outputDescriptors + (index * sizeof(ulong)), out var descriptorAddress) || + descriptorAddress == 0) + { + continue; + } + + if (!ctx.Memory.TryRead(descriptorAddress, descriptor)) + { + continue; + } + + var samples = BinaryPrimitives.ReadUInt32LittleEndian(descriptor); + var channels = BinaryPrimitives.ReadUInt32LittleEndian(descriptor[4..]); + var buffersAddress = BinaryPrimitives.ReadUInt64LittleEndian(descriptor[16..]); + if (samples == 0 || channels == 0 || channels > 64 || buffersAddress == 0 || samples > 65536) + { + continue; + } + + for (uint channel = 0; channel < channels; channel++) + { + if (!ctx.TryReadUInt64(buffersAddress + (channel * sizeof(ulong)), out var bufferAddress) || + bufferAddress == 0) + { + continue; + } + + ClearGuestMemory(ctx, bufferAddress, (ulong)samples * sizeof(float)); + } + } + } + + // The batch record still advances when the job is recorded. The actual + // convolution remains unimplemented, but stale output must not be played. + return AdvanceBatchInfo(ctx, 1024); + } [SysAbiExport( Nid = "9nLbWmRDpa8", @@ -273,6 +316,29 @@ public static class AcmExports return ctx.SetReturn(OrbisGen2Result.ORBIS_GEN2_OK); } + private static ulong ReadStackArg64(CpuContext ctx, int index) + { + var address = ctx[CpuRegister.Rsp] + sizeof(ulong) + ((ulong)index * sizeof(ulong)); + return ctx.TryReadUInt64(address, out var value) ? value : 0; + } + + private static void ClearGuestMemory(CpuContext ctx, ulong address, ulong byteCount) + { + Span zeros = stackalloc byte[256]; + var remaining = byteCount; + while (remaining != 0) + { + var chunk = (int)Math.Min(remaining, (ulong)zeros.Length); + if (!ctx.Memory.TryWrite(address, zeros[..chunk])) + { + return; + } + + address += (ulong)chunk; + remaining -= (ulong)chunk; + } + } + private static void Trace(string message) { if (string.Equals( diff --git a/src/SharpEmu.Libs/Agc/AgcExports.CommandPackets.cs b/src/SharpEmu.Libs/Agc/AgcExports.CommandPackets.cs index 75f378a..e4e9711 100644 --- a/src/SharpEmu.Libs/Agc/AgcExports.CommandPackets.cs +++ b/src/SharpEmu.Libs/Agc/AgcExports.CommandPackets.cs @@ -841,7 +841,10 @@ public static partial class AgcExports var payloadDwords = Math.Max(((uint)marker.Length + 4) / 4, 1); var packetDwords = payloadDwords + 1; - if (!TryAllocateCommandDwords(ctx, commandBufferAddress, packetDwords, out var commandAddress) || + // The returned pointer must name reserved packet storage. Returning an unreserved + // cursor aliases the next packet and also bypasses the buffer-full callback. + // Only a marker from a foreign DCB writer is skipped, with a null packet pointer. + if (!TryAllocateCommandDwords(ctx, commandBufferAddress, packetDwords, out var commandAddress, isMarker: true) || !TryWriteUInt32(ctx, commandAddress, Pm4(packetDwords, ItNop, RPushMarker))) { return ReturnPointer(ctx, 0); @@ -884,7 +887,7 @@ public static partial class AgcExports { var commandBufferAddress = ctx[CpuRegister.Rdi]; if (commandBufferAddress == 0 || - !TryAllocateCommandDwords(ctx, commandBufferAddress, 2, out var commandAddress) || + !TryAllocateCommandDwords(ctx, commandBufferAddress, 2, out var commandAddress, isMarker: true) || !TryWriteUInt32(ctx, commandAddress, Pm4(2, ItNop, RPopMarker)) || !TryWriteUInt32(ctx, commandAddress + 4, 0)) { diff --git a/src/SharpEmu.Libs/Agc/AgcExports.ControlFlow.cs b/src/SharpEmu.Libs/Agc/AgcExports.ControlFlow.cs index 125c2c8..7597523 100644 --- a/src/SharpEmu.Libs/Agc/AgcExports.ControlFlow.cs +++ b/src/SharpEmu.Libs/Agc/AgcExports.ControlFlow.cs @@ -280,18 +280,22 @@ public static partial class AgcExports ExportName = "sceAgcAcbJump", Target = Generation.Gen5, LibraryName = "libSceAgc")] + // (acb, cachePolicy, target, sizeDwords): an ACB jump always chains, so unlike + // sceAgcDcbJump it takes no mode argument. public static int AcbJump(CpuContext ctx) { var acb = ctx[CpuRegister.Rdi]; - var target = ctx[CpuRegister.Rsi]; - var sizeDwords = (uint)ctx[CpuRegister.Rdx]; - if (acb == 0) + var cachePolicy = (uint)ctx[CpuRegister.Rsi]; + var target = ctx[CpuRegister.Rdx]; + var sizeDwords = (uint)ctx[CpuRegister.Rcx]; + if (acb == 0 || cachePolicy > 3) { return ReturnPointer(ctx, 0); } const uint chainMode = 1; var control = 0x0F20_0000u | + ((cachePolicy & 0x3u) << 28) | (chainMode << 20) | (sizeDwords & 0xFFFFFu); if (!TryAllocateCommandDwords(ctx, acb, 4, out var cmd) || diff --git a/src/SharpEmu.Libs/Agc/AgcExports.DrawPackets.cs b/src/SharpEmu.Libs/Agc/AgcExports.DrawPackets.cs index dfb8c60..0317c42 100644 --- a/src/SharpEmu.Libs/Agc/AgcExports.DrawPackets.cs +++ b/src/SharpEmu.Libs/Agc/AgcExports.DrawPackets.cs @@ -1,6 +1,7 @@ // Copyright (C) 2026 SharpEmu Emulator Project // SPDX-License-Identifier: GPL-2.0-or-later +using System.Collections.Concurrent; using SharpEmu.HLE; using SharpEmu.Libs.Kernel; @@ -133,13 +134,15 @@ public static partial class AgcExports var commandBufferAddress = ctx[CpuRegister.Rdi]; var indexCount = (uint)ctx[CpuRegister.Rsi]; var indexAddress = ctx[CpuRegister.Rdx]; - var modifier = (uint)ctx[CpuRegister.Rcx]; + var modifier = ctx[CpuRegister.Rcx]; - if (commandBufferAddress == 0 || modifier != 0x4000_0000) + if (commandBufferAddress == 0) { return ReturnPointer(ctx, 0); } + NoteDrawModifier("sceAgcDcbDrawIndex", modifier); + if (!TryAllocateCommandDwords(ctx, commandBufferAddress, 5, out var baseCommand) || !TryWriteUInt32(ctx, baseCommand, Pm4(3, ItIndexBase, 0)) || !TryWriteUInt32(ctx, baseCommand + 4, (uint)indexAddress) || @@ -239,11 +242,13 @@ public static partial class AgcExports var commandBufferAddress = ctx[CpuRegister.Rdi]; var indexCount = (uint)ctx[CpuRegister.Rsi]; var modifier = ctx[CpuRegister.Rdx]; - if (commandBufferAddress == 0 || modifier != 0x4000_0000) + if (commandBufferAddress == 0) { return ReturnPointer(ctx, 0); } + NoteDrawModifier("sceAgcDcbDrawIndexAuto", modifier); + if (!TryAllocateCommandDwords(ctx, commandBufferAddress, 7, out var commandAddress) || !TryWriteUInt32(ctx, commandAddress, Pm4(7, ItNop, RDrawIndexAuto)) || !TryWriteUInt32(ctx, commandAddress + 4, indexCount) || @@ -457,4 +462,19 @@ public static partial class AgcExports #pragma warning restore SHEM006 private static long _indirectDrawEmitRejectCount; + + private static readonly ConcurrentDictionary<(string, ulong), byte> _reportedDrawModifiers = new(); + + // The modifier names the shader stage that receives the draw and the user registers the + // command processor fills for indirect draws. A direct draw emits the same packet for every + // modifier; rejecting the unfamiliar ones dropped whole passes, such as Unreal's volume + // rasterization through the geometry stage, without a trace. + private static void NoteDrawModifier(string export, ulong modifier) + { + if (modifier != 0x4000_0000 && _reportedDrawModifiers.TryAdd((export, modifier), 0)) + { + Console.Error.WriteLine( + $"[LOADER][INFO] agc.draw_modifier {export} modifier=0x{modifier:X} stage={(uint)modifier >> 29} flags=0x{(uint)modifier & 7u:X}"); + } + } } diff --git a/src/SharpEmu.Libs/Agc/AgcExports.DriverResources.cs b/src/SharpEmu.Libs/Agc/AgcExports.DriverResources.cs index 0e100c0..437c3e1 100644 --- a/src/SharpEmu.Libs/Agc/AgcExports.DriverResources.cs +++ b/src/SharpEmu.Libs/Agc/AgcExports.DriverResources.cs @@ -325,10 +325,11 @@ public static partial class AgcExports } // Tessellation-factor ring and hull-shader off-chip buffers are guest-driver - // configuration for on-hardware tessellation memory. Our translator handles - // shader execution directly, so there is no guest-side ring to program: the - // guest driver only needs these to report success so init proceeds. Games - // (e.g. Unity titles) call them during GPU setup and stall if unresolved. + // configuration for on-hardware tessellation memory. The hull writes its factors + // through its own buffer descriptor, so the factor ring needs no host state. The + // off-chip parameter decides how many hull groups may run together; it is kept for + // the tessellation draws. Games (e.g. Unity titles) call these during GPU setup and + // stall if unresolved. [SysAbiExport( Nid = "XlNp7jzGiPo", ExportName = "sceAgcDriverSetTFRing", @@ -352,6 +353,7 @@ public static partial class AgcExports TraceAgc( $"agc.driver_set_hs_offchip_param buffer=0x{ctx[CpuRegister.Rdi]:X16} " + $"param=0x{(uint)ctx[CpuRegister.Rsi]:X8}"); + Gpu.TessellationOffchip.Configure((uint)ctx[CpuRegister.Rsi]); return ctx.SetReturn(OrbisGen2Result.ORBIS_GEN2_OK); } } diff --git a/src/SharpEmu.Libs/Agc/AgcExports.Events.cs b/src/SharpEmu.Libs/Agc/AgcExports.Events.cs index d21a8b6..cd71c9c 100644 --- a/src/SharpEmu.Libs/Agc/AgcExports.Events.cs +++ b/src/SharpEmu.Libs/Agc/AgcExports.Events.cs @@ -64,19 +64,29 @@ public static partial class AgcExports var commandBufferAddress = ctx[CpuRegister.Rdi]; var eventType = (uint)(ctx[CpuRegister.Rsi] & 0xFF); var eventAddress = ctx[CpuRegister.Rdx]; - if (commandBufferAddress == 0 || eventType > 0x3F || eventAddress != 0) + if (commandBufferAddress == 0 || eventType > 0x3F || (eventAddress != 0 && (eventType & ~1u) != 0x38)) { return ReturnPointer(ctx, 0); } - if (!TryAllocateCommandDwords(ctx, commandBufferAddress, 2, out var commandAddress) || - !TryWriteUInt32(ctx, commandAddress, Pm4(2, ItEventWrite, 0)) || - !TryWriteUInt32(ctx, commandAddress + 4, eventType)) + // Counter selection and counter dumps carry a payload in the address fields. + var hasAddress = (eventType & ~1u) == 0x38; + var packetDwords = hasAddress ? 4u : 2u; + if (!TryAllocateCommandDwords(ctx, commandBufferAddress, packetDwords, out var commandAddress) || + !TryWriteUInt32(ctx, commandAddress, Pm4(packetDwords, ItEventWrite, 0)) || + !TryWriteUInt32(ctx, commandAddress + 4, hasAddress ? eventType | 0x100u : eventType)) { return ReturnPointer(ctx, 0); } - TraceAgc($"agc.dcb_event_write buf=0x{commandBufferAddress:X16} cmd=0x{commandAddress:X16} type={eventType}"); + if (hasAddress && + (!TryWriteUInt32(ctx, commandAddress + 8, (uint)eventAddress & ~7u) || + !TryWriteUInt32(ctx, commandAddress + 12, (uint)(eventAddress >> 32)))) + { + return ReturnPointer(ctx, 0); + } + + TraceAgc($"agc.dcb_event_write buf=0x{commandBufferAddress:X16} cmd=0x{commandAddress:X16} type={eventType} address=0x{eventAddress:X16}"); return ReturnPointer(ctx, commandAddress); } diff --git a/src/SharpEmu.Libs/Agc/AgcExports.ShaderState.cs b/src/SharpEmu.Libs/Agc/AgcExports.ShaderState.cs index 07dcd96..a01b1d3 100644 --- a/src/SharpEmu.Libs/Agc/AgcExports.ShaderState.cs +++ b/src/SharpEmu.Libs/Agc/AgcExports.ShaderState.cs @@ -223,8 +223,7 @@ public static partial class AgcExports descriptor[(int)ShaderCodeOffset..]); var continuationSize = BinaryPrimitives.ReadUInt32LittleEndian( descriptor[(int)ShaderSizeOffset..]); - if (continuationCodeAddress <= entryCodeAddress || - continuationCodeAddress - entryCodeAddress > uint.MaxValue || + if (continuationCodeAddress == entryCodeAddress || !IsValidDeclaredShaderSize(continuationSize) || !CanReadShaderRange(ctx, continuationCodeAddress, continuationSize)) { @@ -249,7 +248,8 @@ public static partial class AgcExports } } - var distance = continuationCodeAddress - entryCodeAddress; + var distance = continuationCodeAddress > entryCodeAddress + ? continuationCodeAddress - entryCodeAddress : entryCodeAddress - continuationCodeAddress; if (distance >= bestDistance) { continue; diff --git a/src/SharpEmu.Libs/Agc/AgcExports.cs b/src/SharpEmu.Libs/Agc/AgcExports.cs index 89fb747..6a2fa40 100644 --- a/src/SharpEmu.Libs/Agc/AgcExports.cs +++ b/src/SharpEmu.Libs/Agc/AgcExports.cs @@ -488,10 +488,94 @@ public static partial class AgcExports return fingerprint; } - private static bool TryAllocateCommandDwords(CpuContext ctx, ulong commandBufferAddress, uint sizeDwords, out ulong commandAddress) - => TryPrepareCommandDwords(ctx, commandBufferAddress, sizeDwords, true, out commandAddress); + private static bool TryAllocateCommandDwords(CpuContext ctx, ulong commandBufferAddress, uint sizeDwords, out ulong commandAddress, + bool isMarker = false) + => TryPrepareCommandDwords(ctx, commandBufferAddress, sizeDwords, true, out commandAddress, isMarker); - private static bool TryPrepareCommandDwords(CpuContext ctx, ulong commandBufferAddress, uint sizeDwords, bool advanceCursor, out ulong commandAddress) + // Inline guest stores do not take the allocation lock. Remember the last ordinary + // packet writer so a diagnostic marker from another guest thread cannot reserve or + // overwrite that writer's unpublished inline packet (Yotei 1.512). Ordinary exports + // can transfer a buffer to a new writer. Scope identities to memory, not guest addresses + // reused by another game; guest handles survive migration between host threads. + private static readonly ConditionalWeakTable> _commandBufferWriters = new(); + + private readonly record struct CommandBufferWriter(ulong ThreadHandle, CpuContext Context) + { + public bool Matches(CpuContext ctx) => ThreadHandle != 0 && GuestThreadExecution.CurrentGuestThreadHandle != 0 + ? ThreadHandle == GuestThreadExecution.CurrentGuestThreadHandle + : ReferenceEquals(Context, ctx); + } + + // Guest threads run HLE exports concurrently. Two exports appending to one command buffer + // both read its cursor before either stores it back, and the second packet overwrites the + // first (a Yōtei DCB lost the header of a SET_SH_REG to a concurrent PopMarker). The + // cursor update is made atomic per buffer; the lock is not held across the guest's + // buffer-full callback. + private static readonly object[] _commandBufferLocks = CreateCommandBufferLocks(); + private static readonly System.Collections.Concurrent.ConcurrentDictionary _reportedConcurrentCommandBuffers = new(); + + private static object[] CreateCommandBufferLocks() + { + var locks = new object[64]; + for (var index = 0; index < locks.Length; index++) + { + locks[index] = new object(); + } + + return locks; + } + + // The buffer each stripe's holder is appending to, so contention between two buffers that + // share a stripe is not reported as a race on one buffer. + private static readonly ulong[] _commandBufferLockOwners = new ulong[64]; + + private static bool TryPrepareCommandDwords(CpuContext ctx, ulong commandBufferAddress, uint sizeDwords, bool advanceCursor, + out ulong commandAddress, bool isMarker = false) + { + // Worker threads keep their command buffer objects at the same offset of stacks 16 MiB + // apart; a multiplicative hash keeps them on different stripes. + var stripe = (int)((commandBufferAddress * 0x9E3779B97F4A7C15UL) >> 58); + var gate = _commandBufferLocks[stripe]; + if (!Monitor.TryEnter(gate)) + { + if (Volatile.Read(ref _commandBufferLockOwners[stripe]) == commandBufferAddress && + _reportedConcurrentCommandBuffers.TryAdd(commandBufferAddress, 0)) + { + Console.Error.WriteLine( + $"[AGC][WARN] Two threads append to one command buffer: buf=0x{commandBufferAddress:X16} thread={Environment.CurrentManagedThreadId}."); + } + + Monitor.Enter(gate); + } + + try + { + Volatile.Write(ref _commandBufferLockOwners[stripe], commandBufferAddress); + var writers = _commandBufferWriters.GetValue(CanonicalMemory(ctx.Memory), static _ => new()); + if (isMarker && writers.TryGetValue(commandBufferAddress, out var writer) && !writer.Matches(ctx)) + { + commandAddress = 0; + return false; + } + + // A first marker establishes a writer too; an ordinary export explicitly + // takes ownership, including a legitimate handoff between guest threads. + if (!writers.TryGetValue(commandBufferAddress, out writer) || !writer.Matches(ctx) || + writer.ThreadHandle != GuestThreadExecution.CurrentGuestThreadHandle) + writers[commandBufferAddress] = new(GuestThreadExecution.CurrentGuestThreadHandle, ctx); + + return TryPrepareCommandDwordsLocked(ctx, commandBufferAddress, sizeDwords, advanceCursor, gate, + writers, isMarker, out commandAddress); + } + finally + { + Volatile.Write(ref _commandBufferLockOwners[stripe], 0); + Monitor.Exit(gate); + } + } + + private static bool TryPrepareCommandDwordsLocked(CpuContext ctx, ulong commandBufferAddress, uint sizeDwords, bool advanceCursor, object gate, + ConcurrentDictionary writers, bool isMarker, out ulong commandAddress) { commandAddress = 0; if (sizeDwords == 0 || @@ -511,19 +595,31 @@ public static partial class AgcExports var scheduler = GuestThreadExecution.Scheduler; ulong callbackResult = 0; string? callbackError = null; - if (callback == 0 || - scheduler is null || - !scheduler.TryCallGuestFunction( - ctx, - callback, - commandBufferAddress, - (ulong)sizeDwords + reservedDwords, - userData, - 0, - 0, - "agc_command_buffer_full", - out callbackResult, - out callbackError)) + bool called; + // The callback is guest code; it may block or append to this buffer itself. + Monitor.Exit(gate); + try + { + called = callback != 0 && + scheduler is not null && + scheduler.TryCallGuestFunction( + ctx, + callback, + commandBufferAddress, + (ulong)sizeDwords + reservedDwords, + userData, + 0, + 0, + "agc_command_buffer_full", + out callbackResult, + out callbackError); + } + finally + { + Monitor.Enter(gate); + } + + if (!called) { TraceAgc( $"agc.cmd_alloc_callback_failed buf=0x{commandBufferAddress:X16} " + @@ -532,6 +628,11 @@ public static partial class AgcExports return false; } + // The callback ran with the gate released. An ordinary export may have + // handed the DCB to another writer while this marker was waiting for space. + if (isMarker && !writers[commandBufferAddress].Matches(ctx)) + return false; + TraceAgc( $"agc.cmd_alloc_callback_complete buf=0x{commandBufferAddress:X16} " + $"callback=0x{callback:X16} result=0x{callbackResult:X16}"); diff --git a/src/SharpEmu.Libs/Audio/AjmExports.cs b/src/SharpEmu.Libs/Audio/AjmExports.cs index a9e09fb..5a54711 100644 --- a/src/SharpEmu.Libs/Audio/AjmExports.cs +++ b/src/SharpEmu.Libs/Audio/AjmExports.cs @@ -419,24 +419,12 @@ public static class AjmExports return ctx.SetReturn(OrbisAjmErrorInvalidParameter); } - try - { - var config = new LibAtrac9.Atrac9Config(configData.ToArray()); - Span info = stackalloc byte[20]; - BinaryPrimitives.WriteUInt32LittleEndian(info[0..], unchecked((uint)config.ChannelCount)); - BinaryPrimitives.WriteUInt32LittleEndian(info[4..], unchecked((uint)config.SampleRate)); - BinaryPrimitives.WriteUInt32LittleEndian(info[8..], unchecked((uint)config.FrameSamples)); - BinaryPrimitives.WriteUInt32LittleEndian(info[12..], unchecked((uint)config.SuperframeSamples)); - BinaryPrimitives.WriteUInt32LittleEndian(info[16..], unchecked((uint)config.SuperframeBytes)); - return ctx.SetReturn( - ctx.Memory.TryWrite(outputAddress, info) - ? 0 - : OrbisAjmErrorInvalidParameter); - } - catch (InvalidDataException) - { - return ctx.SetReturn(OrbisAjmErrorInvalidParameter); - } + Span info = stackalloc byte[20]; + return ctx.SetReturn( + Atrac9DecodeState.TryDescribeConfig(configData, info) && + ctx.Memory.TryWrite(outputAddress, info) + ? 0 + : OrbisAjmErrorInvalidParameter); } [SysAbiExport( @@ -607,6 +595,78 @@ public static class AjmExports return ctx.SetReturn(0); } + /// + /// Enqueues a control job (initialize/reset) on an instance. The sideband input + /// carries the codec configuration; the sideband output starts with the basic + /// result the guest polls after the batch completes. + /// + [SysAbiExport( + Nid = "7FZsbyVRM4U", + ExportName = "sceAjmBatchJobControl", + Target = Generation.Gen4 | Generation.Gen5, + LibraryName = "libSceAjm")] + public static int AjmBatchJobControl(CpuContext ctx) + { + var infoAddress = ctx[CpuRegister.Rdi]; + var instanceId = unchecked((uint)ctx[CpuRegister.Rsi]); + var flags = ctx[CpuRegister.Rdx]; + var inputAddress = ctx[CpuRegister.Rcx]; + var inputSize = ctx[CpuRegister.R8]; + var outputAddress = ctx[CpuRegister.R9]; + var outputSize = ReadStackArg64(ctx, 0); + + if (!TryAppendBatchJob(ctx, infoAddress, AjmJobControlSize)) + { + return ctx.SetReturn(OrbisAjmErrorJobCreation); + } + + var status = Atrac9DecodeState.ResultInvalidParameter; + AjmInstanceState? instance = null; + if (TryGetInstance(instanceId, out instance)) + { + status = Atrac9DecodeState.ResultInvalidParameter; + if (instance.Atrac9 is not null) + { + // The initialize sideband carries the four ATRAC9 config bytes after an + // eight-byte header; the config starts with its 0xFE sync byte. + Span sideband = stackalloc byte[(int)Math.Min(inputSize, 16UL)]; + if (inputAddress != 0 && sideband.Length >= 4 && ctx.Memory.TryRead(inputAddress, sideband)) + { + var configOffset = sideband.Length >= 12 && + (sideband[8] is 0xFE or 0x30) + ? 8 + : 0; + if (sideband[configOffset] is 0xFE or 0x30) + { + var totalSamples = sideband.Length >= 4 + ? BinaryPrimitives.ReadUInt32LittleEndian(sideband) + : 0; + var skipSamples = sideband.Length >= 6 + ? BinaryPrimitives.ReadUInt16LittleEndian(sideband[4..]) + : (ushort)0; + status = instance.Atrac9.TryInitialize( + sideband.Slice(configOffset, 4), + totalSamples, + skipSamples) + ? 0 + : Atrac9DecodeState.ResultInvalidParameter; + } + } + } + } + + if (outputSize >= 8) + { + WriteBasicResult(ctx, outputAddress, status); + } + + Trace( + $"batch_job_control instance=0x{instanceId:X8} flags=0x{flags:X16} " + + $"in=0x{inputAddress:X16}+0x{inputSize:X} out=0x{outputAddress:X16}+0x{outputSize:X} status=0x{status:X8} " + + $"atrac9={(instance?.Atrac9 is null ? "none" : "yes")} input={TraceBytes(ctx, inputAddress, inputSize)}"); + return ctx.SetReturn(0); + } + [SysAbiExport( Nid = "3cAg7xN995U", ExportName = "sceAjmBatchJobGetStatistics", @@ -804,7 +864,15 @@ public static class AjmExports return ctx.SetReturn(0); } - private static int AjmBatchJobDecodeCore(CpuContext ctx, bool multipleFrames) + [SysAbiExport( + Nid = "SJ3i0DXP8vg", + ExportName = "sceAjmBatchJobDecodeSplit", + Target = Generation.Gen5, + LibraryName = "libSceAjm")] + public static int AjmBatchJobDecodeSplit(CpuContext ctx) => + AjmBatchJobDecodeCore(ctx, multipleFrames: true, split: true); + + private static int AjmBatchJobDecodeCore(CpuContext ctx, bool multipleFrames, bool split = false) { var infoAddress = ctx[CpuRegister.Rdi]; var instanceId = unchecked((uint)ctx[CpuRegister.Rsi]); @@ -829,6 +897,19 @@ public static class AjmExports 0, 0); } + else if (split) + { + if (instance.Codec != Atrac9CodecType || instance.Atrac9 is null || + !TryCollectBuffers(ctx, true, inputAddress, inputSize, out var inputs, out var inputLength) || + !TryCollectBuffers(ctx, true, outputAddress, outputSize, out var outputs, out var outputLength)) + { + result = new Atrac9DecodeResult(Atrac9DecodeState.ResultInvalidParameter, 0, 0, 0, 0); + } + else + { + result = DecodeAtrac9Scattered(ctx, instance, inputs, inputLength, outputs, outputLength, multipleFrames); + } + } else if (instance.Mp3 is not null) { // GTA V Enhanced streams menu music through AJM MP3 (codec 0). @@ -1136,9 +1217,10 @@ public static class AjmExports return; } - // Layout is positional and driven purely by the job flags: - // Result, then Stream, Format, GaplessDecode, MFrame, CodecInfo — each - // only present when its flag is set and there is still room. + // The run sideband is positional and driven by the job flags. When the + // stream block is requested, +0x10 is the number of samples written + // since the last control job; the title uses it to end a stream exactly + // at its total. Span sideband = stackalloc byte[ AjmSidebandResultBytes + AjmSidebandStreamBytes + @@ -1149,7 +1231,6 @@ public static class AjmExports BinaryPrimitives.WriteInt32LittleEndian(sideband, result.Status); var offset = AjmSidebandResultBytes; - if ((flags & AjmJobSidebandFlagStream) != 0 && (ulong)(offset + AjmSidebandStreamBytes) <= size) { BinaryPrimitives.WriteInt32LittleEndian(sideband[offset..], result.InputConsumed); @@ -1158,9 +1239,10 @@ public static class AjmExports offset += AjmSidebandStreamBytes; } + var streamInfo = instance?.Atrac9?.Info; if ((flags & AjmJobSidebandFlagFormat) != 0 && (ulong)(offset + AjmSidebandFormatBytes) <= size) { - var channels = config?.ChannelCount ?? instance?.MaxChannels ?? 0; + var channels = config?.ChannelCount ?? streamInfo?.Channels ?? instance?.MaxChannels ?? 0; BinaryPrimitives.WriteUInt32LittleEndian(sideband[offset..], unchecked((uint)channels)); BinaryPrimitives.WriteUInt32LittleEndian(sideband[(offset + 4)..], ChannelMaskFor(channels)); BinaryPrimitives.WriteUInt32LittleEndian(sideband[(offset + 8)..], unchecked((uint)(config?.SampleRate ?? 0))); @@ -1377,6 +1459,12 @@ public static class AjmExports multipleFrames ? sideband : sideband[..24]); } + private static string TraceBytes(CpuContext ctx, ulong address, ulong size) + { + Span bytes = stackalloc byte[(int)Math.Min(size, 32UL)]; + return address != 0 && ctx.Memory.TryRead(address, bytes) ? Convert.ToHexString(bytes) : "-"; + } + private static void WriteBasicResult(CpuContext ctx, ulong resultAddress, int status) { if (resultAddress == 0) diff --git a/src/SharpEmu.Libs/Audio/Atrac9DecodeState.cs b/src/SharpEmu.Libs/Audio/Atrac9DecodeState.cs index 92feeb7..d5ee73b 100644 --- a/src/SharpEmu.Libs/Audio/Atrac9DecodeState.cs +++ b/src/SharpEmu.Libs/Audio/Atrac9DecodeState.cs @@ -1,5 +1,6 @@ // Copyright (C) 2026 SharpEmu Emulator Project -// SPDX-License-Identifier: GPL-2.0-or-later +// Portions adapted from YoteiPC (at9_api.c) by llnternet, GPL-2.0. +// SPDX-License-Identifier: GPL-2.0-only using System.Buffers.Binary; using LibAtrac9; @@ -20,6 +21,14 @@ internal readonly record struct Atrac9DecodeResult( ulong TotalDecodedSamples, uint Frames); +internal readonly record struct Atrac9StreamInfo( + int Channels, + int SampleRate, + int SuperframeBytes, + int FramesPerSuperframe, + int FrameSamples, + int BytesPerFrame); + internal readonly record struct Atrac9Gapless(uint TotalSamples, ushort SkipSamples, ushort SkippedSamples); internal sealed class Atrac9DecodeState @@ -32,6 +41,14 @@ internal sealed class Atrac9DecodeState internal const int ResultCodecError = 0x40000000; private const int MaxContainerHeaderBytes = 8 * 1024; + internal static readonly object DecoderInitGate = new(); + private static readonly int[] SampleRates = + [ + 11025, 12000, 16000, 22050, 24000, 32000, 44100, 48000, + 44100, 48000, 64000, 88200, 96000, 128000, 176400, 192000, + ]; + private static readonly int[] FrameSamplePowers = [6, 6, 7, 7, 7, 8, 8, 8, 6, 6, 7, 7, 7, 8, 8, 8]; + private static readonly int[] StandardChannelCounts = [1, 2, 2, 6, 8, 4]; private enum ContainerScan { @@ -42,6 +59,10 @@ internal sealed class Atrac9DecodeState private readonly object _gate = new(); private Atrac9Decoder? _decoder; + private Atrac9Decoder[]? _extendedDecoders; + private byte[]? _decoderConfig; + private byte[]? _extendedDecoderConfig; + private Atrac9StreamInfo? _info; private byte[]? _configData; private byte[]? _compressed; private short[][]? _planarPcm; @@ -49,8 +70,17 @@ internal sealed class Atrac9DecodeState private int _containerHeaderLength; private int _compressedLength; private ulong _totalDecodedSamples; - private Atrac9Gapless _gaplessInit; - private Atrac9Gapless _gapless; + private ulong _totalSampleLimit; + private ulong _skippedSamples; + // Every sample the gapless window removed: the lead-in and the trimmed tail. + private ulong _droppedSamples; + private byte[]? _superframeOutput; + private ushort _skipSamples; + private bool _decodeErrorLogged; + + // Gapless state from sceAjmBatchJobSetGaplessDecode. It is applied again whenever + // the stream restarts, as the hardware re-arms its skip and sample counts. + private Atrac9Gapless? _gaplessInit; public Atrac9Gapless Gapless { @@ -58,7 +88,13 @@ internal sealed class Atrac9DecodeState { lock (_gate) { - return _gapless; + var remaining = _totalSampleLimit == 0 + ? _gaplessInit?.TotalSamples ?? 0 + : (uint)Math.Min(uint.MaxValue, _totalSampleLimit > _totalDecodedSamples ? _totalSampleLimit - _totalDecodedSamples : 0); + return new Atrac9Gapless( + remaining, + (ushort)(_skipSamples - Math.Min(_skippedSamples, _skipSamples)), + (ushort)Math.Min(ushort.MaxValue, _droppedSamples)); } } } @@ -68,14 +104,26 @@ internal sealed class Atrac9DecodeState lock (_gate) { _gaplessInit = new Atrac9Gapless(totalSamples, skipSamples, 0); - if (reset || _gapless == default) + if (reset || (_totalSampleLimit == 0 && _skipSamples == 0)) { - _gapless = _gaplessInit; + ApplyGaplessNoLock(); } } } - private bool HasSampleLimit => _gaplessInit.TotalSamples is not 0 and not uint.MaxValue; + private void ApplyGaplessNoLock() + { + if (_gaplessInit is not { } gapless) + { + return; + } + + // Zero and uint.MaxValue both leave the stream unbounded. + _totalSampleLimit = gapless.TotalSamples is 0 or uint.MaxValue ? 0 : _totalDecodedSamples + gapless.TotalSamples; + _skipSamples = gapless.SkipSamples; + _skippedSamples = 0; + _droppedSamples = 0; + } public Atrac9Config? Config { @@ -88,7 +136,18 @@ internal sealed class Atrac9DecodeState } } - public bool TryInitialize(ReadOnlySpan configData) + internal Atrac9StreamInfo? Info + { + get + { + lock (_gate) + { + return _info; + } + } + } + + public bool TryInitialize(ReadOnlySpan configData, uint totalSamples = 0, ushort skipSamples = 0) { if (configData.Length < 4) { @@ -97,34 +156,56 @@ internal sealed class Atrac9DecodeState lock (_gate) { - try + ClearNoLock(); + if (!TryParseConfig( + configData, + out var info, + out var decoderConfig, + out var extendedDecoderConfig)) { - var normalizedConfig = configData[..4].ToArray(); - var decoder = new Atrac9Decoder(); - decoder.Initialize(normalizedConfig); - var config = decoder.Config; - - _decoder = decoder; - _configData = normalizedConfig; - _compressed = new byte[config.SuperframeBytes]; - _planarPcm = CreatePcmBuffer(config.ChannelCount, config.SuperframeSamples); - _compressedLength = 0; - _totalDecodedSamples = 0; - _containerHeaderLength = 0; - _gapless = _gaplessInit; - Trace( - $"initialized config={Convert.ToHexString(normalizedConfig)} channels={config.ChannelCount} " + - $"rate={config.SampleRate} frame_samples={config.FrameSamples} " + - $"superframe_samples={config.SuperframeSamples} superframe_bytes={config.SuperframeBytes} " + - $"frames_per_superframe={config.FramesPerSuperframe}"); - return true; - } - catch (Exception exception) when ( - exception is ArgumentException or InvalidDataException or InvalidOperationException) - { - Clear(); return false; } + + _configData = configData[..4].ToArray(); + _info = info; + _decoderConfig = decoderConfig; + _extendedDecoderConfig = extendedDecoderConfig; + _compressed = new byte[info.SuperframeBytes]; + _planarPcm = CreatePcmBuffer(info.Channels, info.FrameSamples * info.FramesPerSuperframe); + _compressedLength = 0; + _totalDecodedSamples = 0; + _totalSampleLimit = totalSamples; + _skipSamples = skipSamples; + _skippedSamples = 0; + _droppedSamples = 0; + if (totalSamples == 0 && skipSamples == 0) + { + ApplyGaplessNoLock(); + } + + try + { + lock (DecoderInitGate) + { + InitializeDecodersNoLock(); + } + } + catch (Exception exception) when ( + exception is ArgumentException or InvalidDataException or InvalidOperationException or IndexOutOfRangeException) + { + _decoder = null; + _extendedDecoders = null; + Trace($"decoder_unavailable config={Convert.ToHexString(_configData)} error={exception.GetType().Name}"); + } + + _containerHeaderLength = 0; + Trace( + $"initialized config={Convert.ToHexString(_configData)} channels={info.Channels} " + + $"rate={info.SampleRate} frame_samples={info.FrameSamples} " + + $"superframe_samples={info.FrameSamples * info.FramesPerSuperframe} " + + $"superframe_bytes={info.SuperframeBytes} frames_per_superframe={info.FramesPerSuperframe} " + + $"decoder={(_decoder is not null || _extendedDecoders is not null ? "yes" : "silence")}"); + return true; } } @@ -138,13 +219,16 @@ internal sealed class Atrac9DecodeState return; } - var decoder = new Atrac9Decoder(); - decoder.Initialize((byte[])_configData.Clone()); - _decoder = decoder; + lock (DecoderInitGate) + { + InitializeDecodersNoLock(); + } _compressedLength = 0; _totalDecodedSamples = 0; + _skippedSamples = 0; + _droppedSamples = 0; _containerHeaderLength = 0; - _gapless = _gaplessInit; + ApplyGaplessNoLock(); if (_compressed is not null) { Array.Clear(_compressed); @@ -161,7 +245,7 @@ internal sealed class Atrac9DecodeState { lock (_gate) { - if (_decoder is null || _compressed is null || _planarPcm is null) + if (_info is not { } info || _compressed is null || _planarPcm is null) { return new Atrac9DecodeResult( ResultNotInitialized, @@ -171,8 +255,7 @@ internal sealed class Atrac9DecodeState 0); } - var config = _decoder.Config; - var channels = requestedChannels > 0 ? requestedChannels : config.ChannelCount; + var channels = requestedChannels > 0 ? requestedChannels : info.Channels; if (channels is < 1 or > 16) { return new Atrac9DecodeResult( @@ -184,15 +267,17 @@ internal sealed class Atrac9DecodeState } var bytesPerSample = GetBytesPerSample(encoding); + var outputBytesPerSuperframe = checked(info.FrameSamples * info.FramesPerSuperframe * channels * bytesPerSample); var consumed = 0; var written = 0; uint frames = 0; var status = 0; - while (_compressedLength == config.SuperframeBytes || + while (_compressedLength == info.SuperframeBytes || consumed < input.Length) { - if (HasSampleLimit && _gapless.TotalSamples == 0) + // A stream that reached its gapless total takes no more input. + if (_totalSampleLimit != 0 && _totalDecodedSamples >= _totalSampleLimit) { break; } @@ -223,73 +308,73 @@ internal sealed class Atrac9DecodeState } } - if (_compressedLength < config.SuperframeBytes) + if (_compressedLength < info.SuperframeBytes) { - var copied = Math.Min(config.SuperframeBytes - _compressedLength, input.Length - consumed); + var copied = Math.Min(info.SuperframeBytes - _compressedLength, input.Length - consumed); input.Slice(consumed, copied).CopyTo(_compressed.AsSpan(_compressedLength)); _compressedLength += copied; consumed += copied; } - if (_compressedLength < config.SuperframeBytes) + if (_compressedLength < info.SuperframeBytes) { status |= ResultPartialInput; break; } - var skip = Math.Min(config.SuperframeSamples, (int)_gapless.SkipSamples); - var samples = config.SuperframeSamples - skip; - if (HasSampleLimit) - { - samples = (int)Math.Min((uint)samples, _gapless.TotalSamples); - } - - var outputBytes = checked(samples * channels * bytesPerSample); - if (output.Length - written < outputBytes) + // The gapless window trims the superframe: only the kept samples need room. + var fullSamples = info.FrameSamples * info.FramesPerSuperframe; + var dropSamples = _skippedSamples < _skipSamples + ? (int)Math.Min((ulong)fullSamples, _skipSamples - _skippedSamples) + : 0; + var availableSamples = fullSamples - dropSamples; + var writableSamples = _totalSampleLimit == 0 + ? availableSamples + : (int)Math.Min((ulong)availableSamples, _totalSampleLimit > _totalDecodedSamples + ? _totalSampleLimit - _totalDecodedSamples + : 0); + var sampleBytes = checked(channels * bytesPerSample); + if (output.Length - written < writableSamples * sampleBytes) { status |= ResultNotEnoughRoom; break; } - try + if (_superframeOutput is null || _superframeOutput.Length < outputBytesPerSuperframe) { - _decoder.Decode(_compressed, _planarPcm); - } - catch (Exception exception) when ( - exception is ArgumentException or InvalidDataException or InvalidOperationException or IndexOutOfRangeException) - { - Trace( - $"decode_failed superframe_bytes={config.SuperframeBytes} " + - $"config={Convert.ToHexString(_configData ?? [])} " + - $"head={Convert.ToHexString(_compressed.AsSpan(0, Math.Min(16, _compressed.Length)))} " + - $"error={exception.GetType().Name}: {exception.Message}"); - _compressedLength = 0; - return new Atrac9DecodeResult( - status | ResultInvalidData | ResultCodecError, - consumed, - written, - _totalDecodedSamples, - frames); + _superframeOutput = new byte[outputBytesPerSuperframe]; } - WriteInterleaved( - _planarPcm, - output.Slice(written, outputBytes), - skip, - samples, - channels, - encoding); + var superframe = _superframeOutput.AsSpan(0, outputBytesPerSuperframe); + var decoded = TryDecodeSuperframeNoLock(channels, encoding, superframe); + if (!decoded) + { + if (!_decodeErrorLogged) + { + Trace( + $"decode_failed superframe_bytes={info.SuperframeBytes} " + + $"config={Convert.ToHexString(_configData ?? [])} " + + $"head={Convert.ToHexString(_compressed.AsSpan(0, Math.Min(16, _compressed.Length)))} " + + "playing_silence"); + _decodeErrorLogged = true; + } + superframe.Clear(); + lock (DecoderInitGate) + { + InitializeDecodersNoLock(); + } + } - _gapless = new Atrac9Gapless( - HasSampleLimit ? _gapless.TotalSamples - (uint)samples : _gapless.TotalSamples, - (ushort)(_gapless.SkipSamples - skip), - (ushort)Math.Min(ushort.MaxValue, _gapless.SkippedSamples + config.SuperframeSamples - samples)); - written += outputBytes; + superframe.Slice(dropSamples * sampleBytes, writableSamples * sampleBytes) + .CopyTo(output.Slice(written, writableSamples * sampleBytes)); + _skippedSamples += (ulong)dropSamples; + _droppedSamples += (ulong)(fullSamples - writableSamples); + written += writableSamples * sampleBytes; _compressedLength = 0; - _totalDecodedSamples += unchecked((uint)samples); - frames += unchecked((uint)config.FramesPerSuperframe); + _totalDecodedSamples += unchecked((uint)writableSamples); + frames += unchecked((uint)info.FramesPerSuperframe); - if (!multipleFrames) + if (!multipleFrames || (_totalSampleLimit != 0 && _totalDecodedSamples >= _totalSampleLimit)) { break; } @@ -389,6 +474,24 @@ internal sealed class Atrac9DecodeState return result; } + internal static bool TryDescribeConfig(ReadOnlySpan configData, Span info) + { + lock (DecoderInitGate) + { + if (info.Length < 20 || !TryParseConfig(configData, out var streamInfo, out _, out _)) + { + return false; + } + + BinaryPrimitives.WriteUInt32LittleEndian(info[0..], unchecked((uint)streamInfo.SuperframeBytes)); + BinaryPrimitives.WriteUInt32LittleEndian(info[4..], unchecked((uint)streamInfo.FramesPerSuperframe)); + BinaryPrimitives.WriteUInt32LittleEndian(info[8..], unchecked((uint)streamInfo.Channels)); + BinaryPrimitives.WriteUInt32LittleEndian(info[12..], unchecked((uint)streamInfo.FrameSamples)); + BinaryPrimitives.WriteUInt32LittleEndian(info[16..], unchecked((uint)streamInfo.BytesPerFrame)); + return true; + } + } + private static int GetBytesPerSample(Atrac9PcmEncoding encoding) => encoding switch { @@ -401,13 +504,12 @@ internal sealed class Atrac9DecodeState private static void WriteInterleaved( short[][] source, Span destination, - int firstSample, int samples, int channels, Atrac9PcmEncoding encoding) { var offset = 0; - for (var sample = firstSample; sample < firstSample + samples; sample++) + for (var sample = 0; sample < samples; sample++) { for (var channel = 0; channel < channels; channel++) { @@ -444,9 +546,212 @@ internal sealed class Atrac9DecodeState } } + private void InitializeDecodersNoLock() + { + _decoder = null; + _extendedDecoders = null; + if (_decoderConfig is not null) + { + var decoder = new Atrac9Decoder(); + decoder.Initialize((byte[])_decoderConfig.Clone()); + _decoder = decoder; + } + else if (_extendedDecoderConfig is not null && _info is { } info) + { + var decoders = new Atrac9Decoder[info.Channels]; + for (var index = 0; index < decoders.Length; index++) + { + decoders[index] = new Atrac9Decoder(); + decoders[index].Initialize((byte[])_extendedDecoderConfig.Clone()); + } + + _extendedDecoders = decoders; + } + } + + private bool TryDecodeSuperframeNoLock(int requestedChannels, Atrac9PcmEncoding encoding, Span output) + { + if (_info is not { } info || _planarPcm is null || _compressed is null) + { + return false; + } + + try + { + if (_decoder is not null) + { + _decoder.Decode(_compressed, _planarPcm); + } + else if (_extendedDecoders is not null) + { + if (!TryDecodeExtendedNoLock(info)) + { + return false; + } + } + else + { + output.Clear(); + return true; + } + + WriteInterleaved( + _planarPcm, + output, + info.FrameSamples * info.FramesPerSuperframe, + requestedChannels, + encoding); + return true; + } + catch (Exception exception) when ( + exception is ArgumentException or InvalidDataException or InvalidOperationException or IndexOutOfRangeException) + { + Trace($"decode_exception error={exception.GetType().Name}: {exception.Message}"); + return false; + } + } + + private bool TryDecodeExtendedNoLock(Atrac9StreamInfo info) + { + // Multichannel block ordering and consumed-byte traversal follow + // YoteiPC (llnternet) at9_api.c, GPL-2.0, adapted to the stateful LibAtrac9 API. + if (_extendedDecoders is null || _planarPcm is null || _compressed is null) + { + return false; + } + + var frames = info.FramesPerSuperframe; + var position = 0; + + for (var frame = 0; frame < frames; frame++) + { + for (var channel = 0; channel < info.Channels; channel++) + { + if (!TryBeginExtendedBlock(_compressed, frame, ref position)) + { + return false; + } + + var remaining = _compressed.AsSpan(position).ToArray(); + var used = _extendedDecoders[channel].DecodeFrame( + remaining, + _planarPcm[channel..(channel + 1)], + frame * info.FrameSamples, + frame); + if (used <= 0 || used > remaining.Length) + { + return false; + } + + position += used; + } + } + + return true; + } + + internal static bool TryBeginExtendedBlock(ReadOnlySpan compressed, int frame, ref int position) + { + if (frame < 0 || position < 0 || position > compressed.Length) + { + return false; + } + + if (frame != 0) + { + while (position < compressed.Length && (compressed[position] & 0x80) == 0) + { + position++; + } + } + + return position < compressed.Length; + } + + private static bool TryParseConfig( + ReadOnlySpan configData, + out Atrac9StreamInfo info, + out byte[]? decoderConfig, + out byte[]? extendedDecoderConfig) + { + info = default; + decoderConfig = null; + extendedDecoderConfig = null; + if (configData.Length < 4) + { + return false; + } + + var c = configData[..4]; + var sampleRateIndex = c[1] >> 4; + if (sampleRateIndex >= SampleRates.Length) + { + return false; + } + + if (c[0] == 0xFE) + { + var frameBytes = (((c[2] << 8) | c[3]) >> 5) + 1; + var frames = 1 << ((c[3] >> 3) & 3); + var layout = (c[1] >> 1) & 7; + var channels = layout <= 5 ? StandardChannelCounts[layout] : 2; + var frameSamples = 1 << FrameSamplePowers[sampleRateIndex]; + var normalized = c.ToArray(); + if (layout == 7) + { + normalized[1] = (byte)((normalized[1] & ~0x0E) | (2 << 1)); + } + else if (layout > 5) + { + normalized = null!; + } + + info = new Atrac9StreamInfo( + channels, + SampleRates[sampleRateIndex], + checked(frameBytes * frames), + frames, + frameSamples, + frameBytes); + decoderConfig = normalized; + return true; + } + + if (c[0] != 0x30 || (c[2] & 0xC0) != 0xC0) + { + return false; + } + + var channelsExtended = ((c[1] & 0x0F) + 1) * 4; + var frameBytesExtended = (((c[2] & 7) << 6) | (c[3] >> 2)) + 1; + var sfIndex = c[3] & 3; + var framesExtended = 1 << sfIndex; + var frameSamplesExtended = 1 << FrameSamplePowers[sampleRateIndex]; + var field = ((frameBytesExtended - 1) << 5) | (sfIndex << 3); + extendedDecoderConfig = + [ + 0xFE, + (byte)(sampleRateIndex << 4), + (byte)(field >> 8), + (byte)field, + ]; + info = new Atrac9StreamInfo( + channelsExtended, + SampleRates[sampleRateIndex], + checked(channelsExtended * frameBytesExtended * framesExtended), + framesExtended, + frameSamplesExtended, + frameBytesExtended); + return true; + } + private void Clear() { _decoder = null; + _extendedDecoders = null; + _decoderConfig = null; + _extendedDecoderConfig = null; + _info = null; _configData = null; _compressed = null; _planarPcm = null; @@ -454,5 +759,12 @@ internal sealed class Atrac9DecodeState _containerHeaderLength = 0; _compressedLength = 0; _totalDecodedSamples = 0; + _totalSampleLimit = 0; + _skippedSamples = 0; + _droppedSamples = 0; + _skipSamples = 0; + _decodeErrorLogged = false; } + + private void ClearNoLock() => Clear(); } diff --git a/src/SharpEmu.Libs/Audio/AudioOut2Exports.cs b/src/SharpEmu.Libs/Audio/AudioOut2Exports.cs index 4d3d2d5..ca51fb8 100644 --- a/src/SharpEmu.Libs/Audio/AudioOut2Exports.cs +++ b/src/SharpEmu.Libs/Audio/AudioOut2Exports.cs @@ -113,6 +113,7 @@ public static class AudioOut2Exports private sealed class ContextState { private readonly object _paceGate = new(); + private IHostAudioStream? _backend; private long _nextAdvanceTimestamp; public ContextState(ulong handle, uint frequency, uint grainSamples, uint queueDepth, IHostAudioStream? backend) @@ -121,14 +122,56 @@ public static class AudioOut2Exports Frequency = frequency == 0 ? 48000 : frequency; GrainSamples = grainSamples == 0 ? 256 : grainSamples; QueueDepth = queueDepth == 0 ? 4 : queueDepth; - Backend = backend; + _backend = backend; } public ulong Handle { get; } public uint Frequency { get; } public uint GrainSamples { get; } public uint QueueDepth { get; } - public IHostAudioStream? Backend { get; } + public IHostAudioStream? Backend => Volatile.Read(ref _backend); + + public void AttachBackend(IHostAudioStream? backend) => Volatile.Write(ref _backend, backend); + + public uint QueuedGrains + { + get + { + if (Backend is { } backend) + { + var queuedMilliseconds = backend.QueuedMilliseconds; + if (queuedMilliseconds >= 0) + { + var grainMilliseconds = (double)GrainSamples * 1000.0 / Frequency; + return Math.Min( + QueueDepth, + checked((uint)Math.Ceiling(queuedMilliseconds / grainMilliseconds))); + } + } + + // A context can be intentionally backend-less (headless runs, + // unavailable host audio, or before lazy backend binding). In + // that case PaceAdvance is the queue: report the portion of + // its software clock that has not elapsed yet instead of + // claiming that every grain is free. + lock (_paceGate) + { + var remaining = _nextAdvanceTimestamp - Stopwatch.GetTimestamp(); + if (remaining <= 0) + { + return 0; + } + + var grainTicks = checked( + (long)Math.Ceiling(Stopwatch.Frequency * (double)GrainSamples / Frequency)); + return Math.Min( + QueueDepth, + checked((uint)Math.Ceiling((double)remaining / grainTicks))); + } + } + } + + public uint FreeGrains => QueueDepth - Math.Min(QueueDepth, QueuedGrains); public long GrainTicks => (long)Math.Ceiling(Stopwatch.Frequency * (double)GrainSamples / Frequency); @@ -512,9 +555,14 @@ public static class AudioOut2Exports } else { - // Push is the blocking point of a grain: it waits for the grain an Advance claimed, or - // claims one itself when the title does not advance separately. - ContextState.SleepUntil(context.QueueSlotFree(advancedGrain != 0 ? advancedGrain : context.ReserveGrain())); + // Push queues the grain an Advance claimed, or claims one itself when the title does not + // advance separately. A blocking push waits for its slot; a nonblocking one returns at + // once, as on the hardware, and the grain still counts in the queue level. + var grain = advancedGrain != 0 ? advancedGrain : context.ReserveGrain(); + if (blocking != 0) + { + ContextState.SleepUntil(context.QueueSlotFree(grain)); + } } return SetReturn(ctx, 0); @@ -833,6 +881,10 @@ public static class AudioOut2Exports return SetReturn(ctx, 0); } + // Fixed-size connected state. Do not trust r8/r9 for byte counts. + // Both Demon's Souls and Yotei derive their speaker layout from numChannels + // at +2. Yotei's startup poll waits for that count to become nonzero; it is + // not an independent active flag. +3 is padding, as in the struct above. [SysAbiExport( Nid = "gatEUKG+Ea4", ExportName = "sceAudioOut2PortGetState", @@ -860,6 +912,10 @@ public static class AudioOut2Exports Span state = stackalloc byte[PortStateSize]; state.Clear(); + // +0x00 u16 output = CONNECTED_PRIMARY (1) + // +0x02 u8 channels = from port format when known, else 2 + // +0x03 u8 padding = 0 + // +0x04 s16 volume = 127 (full volume) byte channels = 2; if (Ports.TryGetValue(portHandle, out var port) && TryDecodeDataFormat(port.DataFormat, out var decodedChannels, out _, out _)) @@ -1238,6 +1294,7 @@ public static class AudioOut2Exports } backendName = PrimaryBackendName; + context.AttachBackend(PrimaryBackend); return PrimaryBackend; } @@ -1269,6 +1326,7 @@ public static class AudioOut2Exports } backendName = SecondaryBackendName; + context.AttachBackend(SecondaryBackend); return SecondaryBackend; } } diff --git a/src/SharpEmu.Libs/Audio/AudioOutExports.cs b/src/SharpEmu.Libs/Audio/AudioOutExports.cs index e6ebe19..7e7520a 100644 --- a/src/SharpEmu.Libs/Audio/AudioOutExports.cs +++ b/src/SharpEmu.Libs/Audio/AudioOutExports.cs @@ -51,7 +51,8 @@ public static class AudioOutExports int bytesPerSample, bool isFloat, bool preservesGuestFormat, - IHostAudioStream? backend) + IHostAudioStream? backend, + string backendName) { UserId = userId; Type = type; @@ -63,6 +64,7 @@ public static class AudioOutExports IsFloat = isFloat; PreservesGuestFormat = preservesGuestFormat; Backend = backend; + BackendName = backendName; } public int UserId { get; } @@ -75,6 +77,7 @@ public static class AudioOutExports public bool IsFloat { get; } public bool PreservesGuestFormat { get; } public IHostAudioStream? Backend { get; } + public string BackendName { get; } public object SubmissionGate { get; } = new(); public volatile float Volume = 1.0f; public int BufferByteLength => @@ -195,7 +198,8 @@ public static class AudioOutExports bytesPerSample, isFloat, preservesGuestFormat, - backend); + backend, + backendName); Console.Error.WriteLine( $"[LOADER][INFO] AudioOut port {handle}: {frequency} Hz, " + $"{channels} ch, {(isFloat ? "float32" : "s16")}, " + @@ -598,7 +602,7 @@ public static class AudioOutExports { var peak = PeakAmplitude(source, port.IsFloat, port.BytesPerSample); Console.Error.WriteLine( - $"[LOADER][TRACE] audioout.output#{n} handle={handle} bytes={source.Length} ch={port.Channels} float={port.IsFloat} vol={port.Volume:F2} peak={peak:F4} backend={(port.Backend is null ? "none" : "coreaudio")}"); + $"[LOADER][TRACE] audioout.output#{n} handle={handle} bytes={source.Length} ch={port.Channels} float={port.IsFloat} vol={port.Volume:F2} peak={peak:F4} backend={(port.Backend is null ? "none" : port.BackendName)}"); } } diff --git a/src/SharpEmu.Libs/Codec/Videodec2Decoder.cs b/src/SharpEmu.Libs/Codec/Videodec2Decoder.cs index 7c3c578..8a61cff 100644 --- a/src/SharpEmu.Libs/Codec/Videodec2Decoder.cs +++ b/src/SharpEmu.Libs/Codec/Videodec2Decoder.cs @@ -61,6 +61,7 @@ internal sealed unsafe class Videodec2Decoder : IDisposable private int _swsSourceHeight; private AVPixelFormat _swsSourceFormat = AVPixelFormat.AV_PIX_FMT_NONE; private bool _disposed; + private bool _disposing; private readonly Channel _workChannel = Channel.CreateBounded(new BoundedChannelOptions(WorkQueueCapacity) @@ -82,7 +83,7 @@ internal sealed unsafe class Videodec2Decoder : IDisposable private readonly Thread _worker; private readonly Thread _scheduler; - // Cancelled (not just completed) on Dispose so both loops stop promptly instead of draining a backlog. + // Kept alive until both threads have finished accepted decode and presentation work. private readonly CancellationTokenSource _workerCts = new(); private readonly object _protocolGate = new(); @@ -205,7 +206,15 @@ internal sealed unsafe class Videodec2Decoder : IDisposable public bool EnqueueAccessUnit(byte[] accessUnit) { - return _workChannel.Writer.TryWrite(accessUnit); + lock (_gate) + { + if (_disposed || _disposing) + { + return false; + } + + return _workChannel.Writer.TryWrite(accessUnit); + } } /// Queues an end-of-stream drain: flush FFmpeg and emit one more buffered picture, if any. @@ -794,24 +803,28 @@ internal sealed unsafe class Videodec2Decoder : IDisposable { lock (_gate) { - if (_disposed) + if (_disposed || _disposing) { return; } - _disposed = true; + _disposing = true; } - // Outside _gate: the worker needs it to finish whatever item it's mid-call on. - _workerCts.Cancel(); + // Stop accepting access units, then finish the work already accepted. The + // scheduler must stay alive while the worker drains its bounded frame queue. + RequestDrain(); _workChannel.Writer.TryComplete(); + _worker.Join(); _frameQueue.Writer.TryComplete(); - _worker.Join(TimeSpan.FromSeconds(2)); - _scheduler.Join(TimeSpan.FromSeconds(2)); + _scheduler.Join(); + _workerCts.Cancel(); _workerCts.Dispose(); lock (_gate) { + _disposed = true; + _disposing = false; if (_swsContext != null) { ffmpeg.sws_freeContext(_swsContext); diff --git a/src/SharpEmu.Libs/Codec/Videodec2Exports.cs b/src/SharpEmu.Libs/Codec/Videodec2Exports.cs index 9f6e03f..70a653b 100644 --- a/src/SharpEmu.Libs/Codec/Videodec2Exports.cs +++ b/src/SharpEmu.Libs/Codec/Videodec2Exports.cs @@ -283,6 +283,11 @@ public static class Videodec2Exports return SetReturn(ctx, Ok); } + // SceVideodec2OutputInfo after the caller's thisSize: frameWidth (+0x10) and frameHeight (+0x18) + // are 32-bit fields; framePitch (+0x14) lies between them. + internal static bool TryWritePictureDimensions(CpuContext ctx, ulong address, uint width, uint height) => + ctx.TryWriteUInt32(address + 0x10, width) && ctx.TryWriteUInt32(address + 0x18, height); + [SysAbiExport( Nid = "NtXRa3dRzU0", ExportName = "sceVideodec2GetPictureInfo", diff --git a/src/SharpEmu.Libs/Gpu/Buffers/BdaFaultProcessor.cs b/src/SharpEmu.Libs/Gpu/Buffers/BdaFaultProcessor.cs index febe4ce..6c2cba2 100644 --- a/src/SharpEmu.Libs/Gpu/Buffers/BdaFaultProcessor.cs +++ b/src/SharpEmu.Libs/Gpu/Buffers/BdaFaultProcessor.cs @@ -4,6 +4,7 @@ using System.Runtime.InteropServices; using SharpEmu.HLE.GpuMemory; using SharpEmu.Libs.Gpu.Scheduling; +using SharpEmu.Libs.Kernel; using SharpEmu.ShaderCompiler.Vulkan; using SharpEmu.Libs.Gpu.Vulkan; using Silk.NET.Vulkan; @@ -11,6 +12,8 @@ using Silk.NET.Vulkan; namespace SharpEmu.Libs.Gpu.Buffers; // Turns the GPU fault bitmap into buffers: a compute pass lists faulted pages per area. +// The fault buffer holds two bitmaps of the same size: pages a shader needed but found +// unmapped, then pages a shader wrote through a device address. The same pass lists each. public sealed unsafe class BdaFaultProcessor : IDisposable { private const int MaxPendingFaults = 8; @@ -27,6 +30,8 @@ public sealed unsafe class BdaFaultProcessor : IDisposable private readonly ulong _faultBufferSize; private readonly GpuBuffer _faultBuffer; private readonly GpuBuffer _downloadBuffer; + private readonly GpuBuffer _writtenDownloadBuffer; + private readonly DescriptorSet[] _writtenSets = new DescriptorSet[MaxPendingFaults]; private readonly GpuBuffer? _traceDownloadBuffer; private bool _traceInitialized; private readonly ulong[] _faultAreas = new ulong[MaxPendingFaults]; @@ -46,10 +51,11 @@ public sealed unsafe class BdaFaultProcessor : IDisposable _pageCount = pageCount; _faultBufferSize = pageCount / 8; _faultBuffer = new GpuBuffer(device, scheduler, GpuBufferUsage.DeviceLocal, 0, GpuBuffer.AllFlags, - _faultBufferSize + (GuestGpuMemoryHook.TraceEnabled ? 32UL : 0UL)); + 2 * _faultBufferSize + (GuestGpuMemoryHook.TraceEnabled ? 32UL : 0UL)); if (GuestGpuMemoryHook.TraceEnabled) _traceDownloadBuffer = new GpuBuffer(device, scheduler, GpuBufferUsage.Download, 0, GpuBuffer.AllFlags, MaxPendingFaults * 256); _downloadBuffer = new GpuBuffer(device, scheduler, GpuBufferUsage.Download, 0, GpuBuffer.AllFlags, MaxPendingFaults * PageFaultAreaSize); + _writtenDownloadBuffer = new GpuBuffer(device, scheduler, GpuBufferUsage.Download, 0, GpuBuffer.AllFlags, MaxPendingFaults * PageFaultAreaSize); var vk = device.Vk; var bindings = stackalloc DescriptorSetLayoutBinding[2]; @@ -117,11 +123,11 @@ public sealed unsafe class BdaFaultProcessor : IDisposable RequireSuccess(created, "Fault-buffer pipeline creation"); // One pre-written set per area replaces push descriptors; the bindings never change. - var poolSize = new DescriptorPoolSize { Type = DescriptorType.StorageBuffer, DescriptorCount = 2 * MaxPendingFaults }; + var poolSize = new DescriptorPoolSize { Type = DescriptorType.StorageBuffer, DescriptorCount = 4 * MaxPendingFaults }; var poolInfo = new DescriptorPoolCreateInfo { SType = StructureType.DescriptorPoolCreateInfo, - MaxSets = MaxPendingFaults, + MaxSets = 2 * MaxPendingFaults, PoolSizeCount = 1, PPoolSizes = &poolSize, }; @@ -144,27 +150,35 @@ public sealed unsafe class BdaFaultProcessor : IDisposable RequireSuccess(vk.AllocateDescriptorSets(device.Device, &allocateInfo, sets), "Fault-buffer descriptor set allocation"); } - var infos = stackalloc DescriptorBufferInfo[2 * MaxPendingFaults]; - var writes = stackalloc WriteDescriptorSet[2 * MaxPendingFaults]; + fixed (DescriptorSet* sets = _writtenSets) + { + RequireSuccess(vk.AllocateDescriptorSets(device.Device, &allocateInfo, sets), "Written-page descriptor set allocation"); + } + + var infos = stackalloc DescriptorBufferInfo[4 * MaxPendingFaults]; + var writes = stackalloc WriteDescriptorSet[4 * MaxPendingFaults]; for (var area = 0; area < MaxPendingFaults; area++) { - infos[2 * area] = new DescriptorBufferInfo(_faultBuffer.Handle, 0, _faultBufferSize); - infos[2 * area + 1] = new DescriptorBufferInfo(_downloadBuffer.Handle, (ulong)area * PageFaultAreaSize, PageFaultAreaSize); - for (uint binding = 0; binding < 2; binding++) + var fault = 4 * area; + infos[fault] = new DescriptorBufferInfo(_faultBuffer.Handle, 0, _faultBufferSize); + infos[fault + 1] = new DescriptorBufferInfo(_downloadBuffer.Handle, (ulong)area * PageFaultAreaSize, PageFaultAreaSize); + infos[fault + 2] = new DescriptorBufferInfo(_faultBuffer.Handle, _faultBufferSize, _faultBufferSize); + infos[fault + 3] = new DescriptorBufferInfo(_writtenDownloadBuffer.Handle, (ulong)area * PageFaultAreaSize, PageFaultAreaSize); + for (uint binding = 0; binding < 4; binding++) { - writes[2 * area + binding] = new WriteDescriptorSet + writes[fault + binding] = new WriteDescriptorSet { SType = StructureType.WriteDescriptorSet, - DstSet = _sets[area], - DstBinding = binding, + DstSet = binding < 2 ? _sets[area] : _writtenSets[area], + DstBinding = binding & 1, DescriptorCount = 1, DescriptorType = DescriptorType.StorageBuffer, - PBufferInfo = infos + 2 * area + binding, + PBufferInfo = infos + fault + binding, }; } } - vk.UpdateDescriptorSets(device.Device, 2 * MaxPendingFaults, writes, 0, null); + vk.UpdateDescriptorSets(device.Device, 4 * MaxPendingFaults, writes, 0, null); } public GpuBuffer FaultBuffer @@ -173,7 +187,7 @@ public sealed unsafe class BdaFaultProcessor : IDisposable { if (_traceDownloadBuffer is not null && !_traceInitialized) { - _faultBuffer.Fill(_faultBufferSize, 32, 0); + _faultBuffer.Fill(2 * _faultBufferSize, 32, 0); _traceInitialized = true; } return _faultBuffer; @@ -192,6 +206,8 @@ public sealed unsafe class BdaFaultProcessor : IDisposable var offset = _currentArea * PageFaultAreaSize; _downloadBuffer.Mapped.Slice((int)offset, (int)PageFaultAreaSize).Clear(); _downloadBuffer.Flush(offset, PageFaultAreaSize); + _writtenDownloadBuffer.Mapped.Slice((int)offset, (int)PageFaultAreaSize).Clear(); + _writtenDownloadBuffer.Flush(offset, PageFaultAreaSize); var preBarrier = new BufferMemoryBarrier2 { @@ -202,7 +218,7 @@ public sealed unsafe class BdaFaultProcessor : IDisposable DstQueueFamilyIndex = Vk.QueueFamilyIgnored, Buffer = _faultBuffer.Handle, Offset = 0, - Size = _faultBufferSize, + Size = 2 * _faultBufferSize, }; var postBarrier = preBarrier; postBarrier.DstAccessMask = AccessFlags2.ShaderWriteBit; @@ -218,6 +234,9 @@ public sealed unsafe class BdaFaultProcessor : IDisposable vk.CmdBindDescriptorSets(command, PipelineBindPoint.Compute, _pipelineLayout, 0, 1, &set, 0, null); var threads = _pageCount / 32; vk.CmdDispatch(command, (uint)((threads + 63) / 64), 1, 1); + var writtenSet = _writtenSets[_currentArea]; + vk.CmdBindDescriptorSets(command, PipelineBindPoint.Compute, _pipelineLayout, 0, 1, &writtenSet, 0, null); + vk.CmdDispatch(command, (uint)((threads + 63) / 64), 1, 1); VulkanSynchronization.PipelineBarrier(vk, command, PipelineStageFlags.ComputeShaderBit, PipelineStageFlags.AllCommandsBit, DependencyFlags.ByRegionBit, 0, null, 1, &postBarrier, 0, null); @@ -226,9 +245,9 @@ public sealed unsafe class BdaFaultProcessor : IDisposable var scanTick = _scheduler.CurrentTick; if (_traceDownloadBuffer is not null) { - _traceDownloadBuffer.CopyFrom(_scheduler.Current, _faultBuffer, _faultBufferSize, area * 256UL, 32, + _traceDownloadBuffer.CopyFrom(_scheduler.Current, _faultBuffer, 2 * _faultBufferSize, area * 256UL, 32, destinationAfter: AccessFlags.HostReadBit); - _faultBuffer.Fill(_faultBufferSize, 32, 0); + _faultBuffer.Fill(2 * _faultBufferSize, 32, 0); } _scheduler.QueueCompletionAction(() => { @@ -245,7 +264,17 @@ public sealed unsafe class BdaFaultProcessor : IDisposable var count = Math.Min((uint)faults[0], MaxPageFaults - 1); for (var index = 1; index <= count; index++) { - _faultRanges.Add(faults[index], _pageSize); + var insideMapping = KernelMemoryCompatExports.TryGetMappedRange(faults[index], out var mappingStart, out var mappingLength); + if (!insideMapping && !_cache.IsGuestMemoryMapped(faults[index], _pageSize)) + { + continue; + } + + var window = insideMapping + ? GuestBufferCache.DeviceAddressFaultSpan(faults[index], _pageSize, mappingStart, mappingLength) + : new GuestSpan(faults[index], _pageSize); + _faultRanges.Add(window.Address, window.Size); + _cache.NoteDeviceAddressFault(window.Address, window.Size, insideMapping); GuestGpuMemoryHook.SelectDeviceFaultTracePage(faults[index]); if (SharpEmu.HLE.GpuMemory.GuestGpuMemoryHook.Traces(faults[index], _pageSize)) SharpEmu.HLE.GpuMemory.GuestGpuMemoryHook.Trace(faults[index], _pageSize, @@ -269,6 +298,15 @@ public sealed unsafe class BdaFaultProcessor : IDisposable SharpEmu.HLE.GpuMemory.GuestGpuMemoryHook.Trace(start, size, $"device-address-fault-prepared scan_tick={scanTick} submission_tick={_scheduler.CurrentTick} registered={_cache.IsRegionRegistered(start, size)}"); }); + + // Written pages beyond the list capacity stay in the bitmap for the next scan. + _writtenDownloadBuffer.Invalidate(offset, PageFaultAreaSize); + var written = MemoryMarshal.Cast(_writtenDownloadBuffer.Mapped.Slice((int)offset, (int)PageFaultAreaSize)); + var writtenCount = Math.Min((uint)written[0], MaxPageFaults - 1); + for (var index = 1; index <= writtenCount; index++) + { + _cache.NoteDeviceAddressWrites(written[index], _pageSize); + } _faultAreas[area] = 0; }); @@ -284,6 +322,7 @@ public sealed unsafe class BdaFaultProcessor : IDisposable vk.DestroyDescriptorPool(_device.Device, _pool, null); vk.DestroyDescriptorSetLayout(_device.Device, _layout, null); _downloadBuffer.Dispose(); + _writtenDownloadBuffer.Dispose(); _traceDownloadBuffer?.Dispose(); _faultBuffer.Dispose(); } diff --git a/src/SharpEmu.Libs/Gpu/Buffers/GpuBuffer.cs b/src/SharpEmu.Libs/Gpu/Buffers/GpuBuffer.cs index fec9cf7..6f8ff3a 100644 --- a/src/SharpEmu.Libs/Gpu/Buffers/GpuBuffer.cs +++ b/src/SharpEmu.Libs/Gpu/Buffers/GpuBuffer.cs @@ -112,6 +112,13 @@ public unsafe class GpuBuffer : IDisposable } } + if (result is Result.ErrorOutOfDeviceMemory or Result.ErrorOutOfHostMemory) + { + vk.DestroyBuffer(device.Device, _handle, null); + _handle = default; + throw new OutOfMemoryException($"vkAllocateMemory({usage}, 0x{size:X} bytes) failed with {result}"); + } + RequireSuccess(result, $"vkAllocateMemory({usage}, 0x{size:X} bytes)"); RequireSuccess(vk.BindBufferMemory(device.Device, _handle, _memory, _memoryOffset), "vkBindBufferMemory"); _allocationSize = _slab?.Size ?? requirements.Size; @@ -132,6 +139,8 @@ public unsafe class GpuBuffer : IDisposable { throw SubmissionScheduler.Fatal("The buffer device address is unavailable."); } + + GpuBufferAddressBook.Added(this, _deviceAddress, scheduler.CurrentTick); } // A dedicated allocation arrives zeroed; keep that for a block another buffer used. @@ -149,6 +158,9 @@ public unsafe class GpuBuffer : IDisposable public bool IsCoherent { get; } + // The host mapping of the whole buffer, or null for memory the host cannot map. + public byte* MappedPointer => _mapped; + public GpuBufferUsage Usage { get; } public ulong CpuAddress { get; } @@ -168,6 +180,12 @@ public unsafe class GpuBuffer : IDisposable public void AddStreamScore(int score) => StreamScore += score; + // The guest range whose CPU-write-hot pages a device-address read copied in the current + // memory visibility generation; the render thread owns these. + public long HotSyncGeneration; + public ulong HotSyncStart; + public ulong HotSyncEnd; + // The highest scheduler tick whose command buffer may write this buffer on the GPU. // A readback of it only has to wait for that tick, not for all queued work. public ulong LastGpuWriteTick { get; private set; } @@ -314,6 +332,11 @@ public unsafe class GpuBuffer : IDisposable } } + if (_deviceAddress != 0) + { + GpuBufferAddressBook.Removed(this, _scheduler.CurrentTick); + } + _device.Vk.DestroyBuffer(_device.Device, _handle, null); if (_slab is { } slab) { @@ -339,6 +362,7 @@ public unsafe class GpuBuffer : IDisposable var (required, preferred, avoided) = usage switch { GpuBufferUsage.DeviceLocal => (MemoryPropertyFlags.None, MemoryPropertyFlags.DeviceLocalBit, MemoryPropertyFlags.None), + GpuBufferUsage.Unified => (MemoryPropertyFlags.HostVisibleBit | MemoryPropertyFlags.HostCoherentBit | MemoryPropertyFlags.DeviceLocalBit, MemoryPropertyFlags.HostCachedBit, MemoryPropertyFlags.None), GpuBufferUsage.Upload => (MemoryPropertyFlags.HostVisibleBit, MemoryPropertyFlags.HostCoherentBit, MemoryPropertyFlags.DeviceLocalBit), GpuBufferUsage.Download => (MemoryPropertyFlags.HostVisibleBit, MemoryPropertyFlags.HostCoherentBit | MemoryPropertyFlags.HostCachedBit, MemoryPropertyFlags.DeviceLocalBit), _ => (MemoryPropertyFlags.HostVisibleBit, MemoryPropertyFlags.HostCoherentBit | MemoryPropertyFlags.DeviceLocalBit, MemoryPropertyFlags.None), @@ -392,6 +416,9 @@ public unsafe class GpuBuffer : IDisposable }; } + // Every memory type the buffer may use is full; the owner can free memory and try again. + internal sealed class OutOfMemoryException(string message) : Exception(message); + private static void RequireSuccess(Result result, string operation) { if (result != Result.Success) diff --git a/src/SharpEmu.Libs/Gpu/Buffers/GpuBufferAddressBook.cs b/src/SharpEmu.Libs/Gpu/Buffers/GpuBufferAddressBook.cs new file mode 100644 index 0000000..5e35203 --- /dev/null +++ b/src/SharpEmu.Libs/Gpu/Buffers/GpuBufferAddressBook.cs @@ -0,0 +1,83 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +namespace SharpEmu.Libs.Gpu.Buffers; + +// Device-address ranges of the live buffers and of the most recently destroyed ones, so a +// device fault report can name the buffer a faulting address belongs (or belonged) to. +internal static class GpuBufferAddressBook +{ + private readonly record struct Entry(ulong DeviceAddress, ulong Size, GpuBufferUsage Usage, ulong CpuAddress, long Serial, ulong CreatedTick, ulong DestroyedTick); + + // A Yōtei transition retires more than 256 buffers before an old GPU address + // faults. Keep a bounded history large enough to identify those older ranges. + private const int DestroyedCapacity = 16384; + + private static readonly object Gate = new(); + private static readonly Dictionary Live = new(ReferenceEqualityComparer.Instance); + private static readonly Entry[] Destroyed = new Entry[DestroyedCapacity]; + private static int _destroyedNext; + private static long _serial; + + public static void Added(GpuBuffer buffer, ulong deviceAddress, ulong tick) + { + lock (Gate) + { + Live[buffer] = new Entry(deviceAddress, buffer.Size, buffer.Usage, buffer.CpuAddress, ++_serial, tick, 0); + } + } + + public static void Removed(GpuBuffer buffer, ulong tick) + { + lock (Gate) + { + if (Live.Remove(buffer, out var entry)) + { + Destroyed[_destroyedNext] = entry with { DestroyedTick = tick }; + _destroyedNext = (_destroyedNext + 1) % DestroyedCapacity; + } + } + } + + // The live and destroyed buffers containing the address, else the live ones nearest to it. + public static string Describe(ulong address) + { + lock (Gate) + { + var text = new System.Text.StringBuilder(); + foreach (var entry in Live.Values.Where(entry => Contains(entry, address))) + { + text.Append($" live:{Format(entry, address)}"); + } + + foreach (var entry in Destroyed.Where(entry => entry.Size != 0 && Contains(entry, address))) + { + text.Append($" destroyed:{Format(entry, address)}"); + } + + if (text.Length == 0) + { + var below = Live.Values.Where(entry => entry.DeviceAddress <= address) + .OrderByDescending(entry => entry.DeviceAddress).Take(1); + var above = Live.Values.Where(entry => entry.DeviceAddress > address) + .OrderBy(entry => entry.DeviceAddress).Take(1); + foreach (var entry in below.Concat(above)) + { + text.Append($" nearest:{Format(entry, address)}"); + } + + var lowest = Live.Values.Select(entry => entry.DeviceAddress).DefaultIfEmpty().Min(); + text.Append($" lowest_live=0x{lowest:X} live_count={Live.Count}"); + } + + return text.ToString(); + } + } + + private static bool Contains(Entry entry, ulong address) => + address >= entry.DeviceAddress && address - entry.DeviceAddress < entry.Size; + + private static string Format(Entry entry, ulong address) => + $"[{entry.Usage} device=0x{entry.DeviceAddress:X}+0x{entry.Size:X} cpu=0x{entry.CpuAddress:X} " + + $"offset={(long)(address - entry.DeviceAddress)} serial={entry.Serial} created_tick={entry.CreatedTick} destroyed_tick={entry.DestroyedTick}]"; +} diff --git a/src/SharpEmu.Libs/Gpu/Buffers/GpuBufferUsage.cs b/src/SharpEmu.Libs/Gpu/Buffers/GpuBufferUsage.cs index 09fc7c3..63dde7c 100644 --- a/src/SharpEmu.Libs/Gpu/Buffers/GpuBufferUsage.cs +++ b/src/SharpEmu.Libs/Gpu/Buffers/GpuBufferUsage.cs @@ -9,4 +9,6 @@ public enum GpuBufferUsage : byte Upload, Download, Stream, + // Device-local memory the host can map: guest buffers on unified-memory devices, read back without a copy. + Unified, } diff --git a/src/SharpEmu.Libs/Gpu/Buffers/GpuDeviceInfo.cs b/src/SharpEmu.Libs/Gpu/Buffers/GpuDeviceInfo.cs index d4c636c..80541e8 100644 --- a/src/SharpEmu.Libs/Gpu/Buffers/GpuDeviceInfo.cs +++ b/src/SharpEmu.Libs/Gpu/Buffers/GpuDeviceInfo.cs @@ -7,7 +7,7 @@ namespace SharpEmu.Libs.Gpu.Buffers; // The device facts the stores need: handles, memory types, limits, format support // and the count of live device-memory allocations made through this object. -public sealed unsafe class GpuDeviceInfo : IImageFormatSupport +public sealed unsafe class GpuDeviceInfo : IImageFormatSupport, IDeviceMemoryAllocator { private PhysicalDeviceMemoryProperties _memoryProperties; // Queried for every draw target from the render thread and from other threads; a hit takes no lock. @@ -16,16 +16,38 @@ public sealed unsafe class GpuDeviceInfo : IImageFormatSupport private readonly System.Collections.Concurrent.ConcurrentDictionary<(Format, ImageType, ImageTiling, ImageUsageFlags, ImageCreateFlags), (Result Result, ImageFormatProperties Properties)> _imageFormatProperties = new(); private int _liveAllocations; private int _peakAllocations; + private long _lastFailedAllocationBytes; + + public ulong LastFailedAllocationBytes => (ulong)Interlocked.Read(ref _lastFailedAllocationBytes); // VK_EXT_image_view_min_lod is enabled, so a view can clamp to a texture descriptor's MIN_LOD. public bool ImageViewMinLodSupported { get; init; } + public bool CustomTwoSampleLocationsSupported { get; init; } - public GpuDeviceInfo(Vk vk, PhysicalDevice physicalDevice, Device device) + public GpuDeviceInfo(Vk vk, PhysicalDevice physicalDevice, Device device, bool memoryBudgetEnabled = false) { Vk = vk; PhysicalDevice = physicalDevice; Device = device; vk.GetPhysicalDeviceMemoryProperties(physicalDevice, out _memoryProperties); + DeviceLocalHeapBytes = CalculateDeviceLocalHeapBytes(); + var queriedMemory = memoryBudgetEnabled + ? QueryDeviceLocalMemory() + : (Budget: 0UL, Usage: 0UL, Available: 0UL); + if (queriedMemory.Budget != 0) + { + DeviceLocalBudgetBytes = queriedMemory.Budget; + DeviceLocalUsageBytes = queriedMemory.Usage; + DeviceLocalAvailableBytes = queriedMemory.Available; + HasMemoryBudget = true; + } + else + { + DeviceLocalBudgetBytes = DeviceLocalHeapBytes; + DeviceLocalUsageBytes = 0; + DeviceLocalAvailableBytes = DeviceLocalHeapBytes; + HasMemoryBudget = false; + } vk.GetPhysicalDeviceProperties(physicalDevice, out var properties); MinUniformBufferOffsetAlignment = Math.Max(properties.Limits.MinUniformBufferOffsetAlignment, 1); MinStorageBufferOffsetAlignment = Math.Max(properties.Limits.MinStorageBufferOffsetAlignment, 1); @@ -34,9 +56,11 @@ public sealed unsafe class GpuDeviceInfo : IImageFormatSupport MaxMemoryAllocationCount = properties.Limits.MaxMemoryAllocationCount; MaxComputeWorkGroupCount = (properties.Limits.MaxComputeWorkGroupCount[0], properties.Limits.MaxComputeWorkGroupCount[1], properties.Limits.MaxComputeWorkGroupCount[2]); Slabs = new GpuMemorySlabs(this); + const FormatFeatureFlags blit = FormatFeatureFlags.BlitSrcBit | FormatFeatureFlags.BlitDstBit; + Images.RenderScalePolicy.ConfigureFormatSupport(format => (GetFormatProperties(format).OptimalTilingFeatures & blit) == blit); } - // Shared chunks the small buffers are carved from; freed at device teardown. + // Shared chunks for small buffers; idle chunks can be returned under pressure. internal GpuMemorySlabs Slabs { get; } public Vk Vk { get; } @@ -45,10 +69,23 @@ public sealed unsafe class GpuDeviceInfo : IImageFormatSupport // created when a second queue family (the async readback queue) reads them. public uint[]? SharedQueueFamilies { get; set; } + // Names a Vulkan object for debuggers (RenderDoc); a no-op unless debug labels are enabled. + public Action? NameObject { get; set; } + public PhysicalDevice PhysicalDevice { get; } public Device Device { get; } + public ulong DeviceLocalHeapBytes { get; } + + public ulong DeviceLocalBudgetBytes { get; private set; } + + public ulong DeviceLocalUsageBytes { get; private set; } + + public ulong DeviceLocalAvailableBytes { get; private set; } + + public bool HasMemoryBudget { get; } + public ulong MinUniformBufferOffsetAlignment { get; } public ulong MinStorageBufferOffsetAlignment { get; } @@ -63,8 +100,88 @@ public sealed unsafe class GpuDeviceInfo : IImageFormatSupport public uint MemoryTypeCount => _memoryProperties.MemoryTypeCount; + private ulong CalculateDeviceLocalHeapBytes() + { + var usedHeaps = new bool[_memoryProperties.MemoryHeapCount]; + fixed (PhysicalDeviceMemoryProperties* properties = &_memoryProperties) + { + var memoryTypes = &properties->MemoryTypes.Element0; + for (var index = 0u; index < properties->MemoryTypeCount; index++) + { + if ((memoryTypes[index].PropertyFlags & MemoryPropertyFlags.DeviceLocalBit) != 0) + { + usedHeaps[(int)memoryTypes[index].HeapIndex] = true; + } + } + + var heaps = &properties->MemoryHeaps.Element0; + ulong total = 0; + for (var index = 0u; index < properties->MemoryHeapCount; index++) + { + if (usedHeaps[(int)index]) + { + total += heaps[index].Size; + } + } + + return total; + } + } + + private (ulong Budget, ulong Usage, ulong Available) QueryDeviceLocalMemory() + { + var budget = new PhysicalDeviceMemoryBudgetPropertiesEXT + { + SType = StructureType.PhysicalDeviceMemoryBudgetPropertiesExt, + }; + var properties = new PhysicalDeviceMemoryProperties2 + { + SType = StructureType.PhysicalDeviceMemoryProperties2, + PNext = &budget, + }; + Vk.GetPhysicalDeviceMemoryProperties2(PhysicalDevice, &properties); + + var usedHeaps = new bool[_memoryProperties.MemoryHeapCount]; + fixed (PhysicalDeviceMemoryProperties* memoryProperties = &_memoryProperties) + { + var memoryTypes = &memoryProperties->MemoryTypes.Element0; + for (var index = 0u; index < memoryProperties->MemoryTypeCount; index++) + { + if ((memoryTypes[index].PropertyFlags & MemoryPropertyFlags.DeviceLocalBit) != 0) + { + usedHeaps[(int)memoryTypes[index].HeapIndex] = true; + } + } + } + + ulong totalBudget = 0; + ulong totalUsage = 0; + ulong totalAvailable = 0; + for (var index = 0u; index < _memoryProperties.MemoryHeapCount; index++) + { + if (usedHeaps[(int)index]) + { + var heapBudget = budget.HeapBudget[index]; + var heapUsage = budget.HeapUsage[index]; + totalBudget = checked(totalBudget + heapBudget); + totalUsage = checked(totalUsage + heapUsage); + totalAvailable = checked(totalAvailable + CalculateAvailableBytes(heapBudget, heapUsage)); + } + } + + return totalBudget != 0 + ? (totalBudget, totalUsage, totalAvailable) + : (0, 0, 0); + } + + internal static ulong CalculateAvailableBytes(ulong budget, ulong usage) => + budget > usage ? budget - usage : 0; + public int LiveAllocations => Volatile.Read(ref _liveAllocations); + // Tests make an allocation of the given size fail as an exhausted device would. + internal Func? AllocationFailure { get; set; } + public int PeakAllocations => Volatile.Read(ref _peakAllocations); public MemoryPropertyFlags GetMemoryTypeFlags(uint index) @@ -105,7 +222,12 @@ public sealed unsafe class GpuDeviceInfo : IImageFormatSupport { fixed (MemoryAllocateInfo* pointer = &info) { - var result = Vk.AllocateMemory(Device, pointer, null, out memory); + memory = default; + var result = AllocationFailure?.Invoke(info.AllocationSize) == true + ? Result.ErrorOutOfDeviceMemory + : Vk.AllocateMemory(Device, pointer, null, out memory); + if (result != Result.Success) + Interlocked.Exchange(ref _lastFailedAllocationBytes, checked((long)info.AllocationSize)); if (result == Result.Success) { var live = Interlocked.Increment(ref _liveAllocations); diff --git a/src/SharpEmu.Libs/Gpu/Buffers/GpuMemorySlabs.cs b/src/SharpEmu.Libs/Gpu/Buffers/GpuMemorySlabs.cs index acf45d1..46bec7e 100644 --- a/src/SharpEmu.Libs/Gpu/Buffers/GpuMemorySlabs.cs +++ b/src/SharpEmu.Libs/Gpu/Buffers/GpuMemorySlabs.cs @@ -21,7 +21,7 @@ internal sealed unsafe class GpuMemorySlabs private readonly GpuDeviceInfo _device; private readonly object _gate = new(); private readonly Dictionary<(uint MemoryType, bool DeviceAddress), Pool> _pools = []; - private readonly List _chunks = []; + private readonly Dictionary _chunks = []; private readonly List _quarantine = []; private int _foreignReaders; @@ -39,6 +39,50 @@ internal sealed unsafe class GpuMemorySlabs Enumerable.Range(0, ClassCount).Select(_ => new Stack<(DeviceMemory, ulong, nint, bool)>()).ToArray(); } + private sealed class Chunk(uint memoryType, bool deviceAddress, int sizeClass) + { + public readonly uint MemoryType = memoryType; + public readonly bool DeviceAddress = deviceAddress; + public readonly int SizeClass = sizeClass; + // Quarantined blocks remain in use until foreign readers have finished. + public int InUse; + } + + internal ulong AllocatedBytes { get { lock (_gate) return (ulong)_chunks.Count * ChunkSize; } } + + private void DrainQuarantine() + { + if (Volatile.Read(ref _foreignReaders) != 0) return; + foreach (var released in _quarantine) + { + var chunk = _chunks[released.Memory]; + chunk.InUse--; + _pools[(released.MemoryType, released.DeviceAddress)].Free[chunk.SizeClass] + .Push((released.Memory, released.Offset, released.MappedAddress, true)); + } + _quarantine.Clear(); + } + + // Only blocks already retired after GPU completion may enter quarantine. + // Under pressure return completely idle chunks, retaining live neighbours. + public void ReleaseUnused() + { + lock (_gate) + { + DrainQuarantine(); + foreach (var memory in _chunks.Where(pair => pair.Value.InUse == 0).Select(pair => pair.Key).ToArray()) + { + var chunk = _chunks[memory]; + var free = _pools[(chunk.MemoryType, chunk.DeviceAddress)].Free[chunk.SizeClass]; + var kept = free.Where(block => block.Memory.Handle != memory.Handle).ToArray(); + free.Clear(); + for (var index = kept.Length - 1; index >= 0; index--) free.Push(kept[index]); + _chunks.Remove(memory); + _device.FreeMemory(memory); + } + } + } + // A readback on the second queue copies from buffers the render thread may retire (and // recycle) while the copy is still pending. A dedicated allocation kept its old bytes // after vkFreeMemory, which hid that race; a recycled block does not. Released blocks @@ -69,17 +113,7 @@ internal sealed unsafe class GpuMemorySlabs var sizeClass = BitOperations.Log2(blockSize) - BitOperations.Log2(MinBlockSize); lock (_gate) { - if (_quarantine.Count != 0 && Volatile.Read(ref _foreignReaders) == 0) - { - foreach (var released in _quarantine) - { - var releasedClass = BitOperations.Log2(released.Size) - BitOperations.Log2(MinBlockSize); - _pools[(released.MemoryType, released.DeviceAddress)].Free[releasedClass] - .Push((released.Memory, released.Offset, released.MappedAddress, true)); - } - - _quarantine.Clear(); - } + DrainQuarantine(); if (!_pools.TryGetValue((memoryType, deviceAddress), out var pool)) { @@ -94,6 +128,7 @@ internal sealed unsafe class GpuMemorySlabs } var (memory, offset, mapped, recycled) = free.Pop(); + _chunks[memory].InUse++; block = new Block(memory, offset, blockSize, mapped, memoryType, deviceAddress, recycled); return true; } @@ -112,7 +147,7 @@ internal sealed unsafe class GpuMemorySlabs { lock (_gate) { - foreach (var chunk in _chunks) + foreach (var chunk in _chunks.Keys) { _device.FreeMemory(chunk); } @@ -155,7 +190,8 @@ internal sealed unsafe class GpuMemorySlabs mapped = (byte*)pointer; } - _chunks.Add(memory); + _chunks.Add(memory, new Chunk(memoryType, deviceAddress, + BitOperations.Log2(blockSize) - BitOperations.Log2(MinBlockSize))); // Push in reverse so blocks come out in address order. for (var offset = ChunkSize - blockSize; ; offset -= blockSize) { diff --git a/src/SharpEmu.Libs/Gpu/Buffers/GuestBufferCache.Watch.cs b/src/SharpEmu.Libs/Gpu/Buffers/GuestBufferCache.Watch.cs new file mode 100644 index 0000000..cb4cda6 --- /dev/null +++ b/src/SharpEmu.Libs/Gpu/Buffers/GuestBufferCache.Watch.cs @@ -0,0 +1,116 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Globalization; +using SharpEmu.HLE.GpuMemory; + +namespace SharpEmu.Libs.Gpu.Buffers; + +// SHARPEMU_WATCH_BLOCKS=addr:len[,addr:len...] (diagnostic): after every presented frame, logs each +// watched range whose bytes changed in guest memory or in the GPU copy, with both contents, and +// logs every cache event that touches a watched range (device-address write reports, uploads, +// writable bindings, CPU write faults). Ticks date each line against the GPU timeline. +public sealed unsafe partial class GuestBufferCache +{ + private readonly record struct WatchedRange(ulong Address, int Length); + + private static readonly WatchedRange[] WatchedRanges = ParseWatchedRanges(Environment.GetEnvironmentVariable("SHARPEMU_WATCH_BLOCKS")); + private const int MaxWatchLines = 20000; + private readonly byte[]?[] _watchedGuest = new byte[WatchedRanges.Length][]; + private readonly byte[]?[] _watchedGpu = new byte[WatchedRanges.Length][]; + private long _watchFrame; + private static int _watchLines; + + private static WatchedRange[] ParseWatchedRanges(string? value) + { + if (string.IsNullOrWhiteSpace(value)) + { + return []; + } + + var ranges = new List(); + foreach (var item in value.Split(',', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries)) + { + var parts = item.Split(':'); + if (ulong.TryParse(parts[0].Replace("0x", "", StringComparison.OrdinalIgnoreCase), NumberStyles.HexNumber, CultureInfo.InvariantCulture, out var address) && + address != 0) + { + var length = parts.Length > 1 && int.TryParse(parts[1].Replace("0x", "", StringComparison.OrdinalIgnoreCase), NumberStyles.HexNumber, CultureInfo.InvariantCulture, out var parsed) + ? Math.Clamp(parsed, 4, 256) + : 64; + ranges.Add(new WatchedRange(address, length)); + } + } + + return [.. ranges]; + } + + private static bool Watches(ulong address, ulong size) + { + foreach (var range in WatchedRanges) + { + if (address < range.Address + (ulong)range.Length && range.Address < address + size) + { + return true; + } + } + + return false; + } + + private void WatchEvent(ulong address, ulong size, string kind) + { + if (WatchedRanges.Length == 0 || !Watches(address, size) || Interlocked.Increment(ref _watchLines) > MaxWatchLines) + { + return; + } + + Console.Error.WriteLine(FormattableString.Invariant( + $"[GPU][WATCH] frame={_watchFrame} event={kind} addr=0x{address:X} size=0x{size:X} submission_tick={_scheduler.CurrentTick} completed_tick={_scheduler.Timeline.CompletedTick}")); + } + + public void WatchFrame() + { + if (WatchedRanges.Length == 0) + { + return; + } + + _watchFrame++; + for (var index = 0; index < WatchedRanges.Length; index++) + { + var range = WatchedRanges[index]; + var guest = new byte[range.Length]; + if (!_backing.TryReadBacking(range.Address, guest)) + { + guest = null; + } + + byte[]? gpu = null; + if (FindOwner(range.Address, (ulong)range.Length) is { MappedPointer: not null, IsCoherent: true } owner) + { + gpu = new ReadOnlySpan(owner.MappedPointer + owner.Offset(range.Address), range.Length).ToArray(); + } + + var guestChanged = !Same(guest, _watchedGuest[index]); + var gpuChanged = !Same(gpu, _watchedGpu[index]); + _watchedGuest[index] = guest; + _watchedGpu[index] = gpu; + if ((!guestChanged && !gpuChanged) || Interlocked.Increment(ref _watchLines) > MaxWatchLines) + { + continue; + } + + var page = range.Address & ~(TrackerLayout.PageBytes - 1); + var cpuDirty = _tracker.HasCpuDirtyPages(page, TrackerLayout.PageBytes); + var gpuDirty = _tracker.HasGpuDirtyPages(page, TrackerLayout.PageBytes); + var guestHex = guest is null ? "n/a" : Convert.ToHexString(guest); + var gpuHex = gpu is null ? "n/a" : Convert.ToHexString(gpu); + Console.Error.WriteLine(FormattableString.Invariant( + $"[GPU][WATCH] frame={_watchFrame} range=0x{range.Address:X} guest_changed={guestChanged} gpu_changed={gpuChanged} cpu_dirty={cpuDirty} gpu_dirty={gpuDirty} submission_tick={_scheduler.CurrentTick} completed_tick={_scheduler.Timeline.CompletedTick} guest={guestHex} gpu={gpuHex}")); + } + } + + private static bool Same(byte[]? left, byte[]? right) => + left is null ? right is null : right is not null && left.AsSpan().SequenceEqual(right); +} diff --git a/src/SharpEmu.Libs/Gpu/Buffers/GuestBufferCache.cs b/src/SharpEmu.Libs/Gpu/Buffers/GuestBufferCache.cs index 183f607..0f41812 100644 --- a/src/SharpEmu.Libs/Gpu/Buffers/GuestBufferCache.cs +++ b/src/SharpEmu.Libs/Gpu/Buffers/GuestBufferCache.cs @@ -18,7 +18,7 @@ public readonly record struct DownloadPiece(GpuBuffer Buffer, ulong SourceOffset public readonly record struct OverlapSpan(int First, int Last, ulong Begin, ulong End, bool HasStreamLeap); // Guest memory mirrored in device buffers: upload on use, download on CPU fault, page table for BDA. -public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable +public sealed unsafe partial class GuestBufferCache : IGuestBufferStore, IDisposable { public const int CachingPageBits = 14; public const ulong CachingPageSize = 1UL << CachingPageBits; @@ -41,7 +41,13 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable private readonly SubmissionScheduler _scheduler; private readonly IGpuQueueRelay _relay; private readonly GuestBufferUploader _uploader; + private readonly bool _unifiedBuffers; private readonly IGuestBackedSpace _backing; + private readonly ICpuMemory _guest; + + // False for addresses the guest never mapped. A shader that follows a bad pointer there + // reads zeros, as on the console; caching that memory only fills VRAM with junk buffers. + internal bool IsGuestMemoryMapped(ulong guestAddress, ulong size) => _guest.CanRead(guestAddress, size); private readonly BdaFaultProcessor _faults; private readonly GpuBuffer _gds; private readonly GpuBuffer _bdaPageTable; @@ -71,7 +77,9 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable _scheduler = scheduler; _relay = relay; _backing = backing; + _guest = guest; _faults = new BdaFaultProcessor(device, scheduler, this, CachingPageBits, CachingPageCount); + _unifiedBuffers = UnifiedBuffersEnabled && HasUnifiedMemoryType(device); _gds = new GpuBuffer(device, scheduler, GpuBufferUsage.Stream, 0, GpuBuffer.AllFlags, GdsBufferSize); _bdaPageTable = new GpuBuffer(device, scheduler, GpuBufferUsage.DeviceLocal, 0, GpuBuffer.AllFlags, BdaPageTableSize); _tracker = new GuestPageTracker(pages); @@ -98,7 +106,9 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable public GpuBuffer FaultBuffer => _faults.FaultBuffer; - public ulong TotalUsedMemory => _registry.RegisteredBytes; + public ulong TotalUsedMemory => _registry.RegisteredBytes + _registry.RetiredBytes; + + public bool RetirementOverBudget => _registry.RetiredBytes > 256UL * MiB; public int BufferCount => _registry.RegisteredCount; @@ -127,6 +137,7 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable // A CPU write fault: true when the range is tracked and any GPU data reached guest memory. bool IGuestBufferStore.MarkCpuWrite(ulong address, ulong size) { + WatchEvent(address, size, "cpu-write-fault"); var tracked = _tracker.InvalidateRegion(address, size, out var needsGpuFlush); var completed = !needsGpuFlush || ReadMemoryOrAwaitShutdown(address, size, isWrite: true, GuestMemoryProfile.ReadbackSource.CpuWriteInvalidation); @@ -198,7 +209,164 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable return CreateBuffer(guestAddress, size); } - public (GpuBuffer Buffer, ulong Offset) ObtainBuffer(ulong guestAddress, ulong size, bool isWritten, bool isTexelBuffer = false, ResourceSlotIdentifier bufferIdentifier = default) + public void PrepareBufferAllocations(ReadOnlySpan ranges) + { + // Draws have few ranges; keep that common case off the managed heap. + Span pages = ranges.Length <= 64 ? stackalloc GuestSpan[ranges.Length] : new GuestSpan[ranges.Length]; + for (var index = 0; index < ranges.Length; index++) + { + var range = ranges[index]; + if (!IsValidRange(range.Address, range.Size) || + !PageOwnerTable.TryGetPageRange(range.Address, range.Size, out var first, out var end)) + throw SubmissionScheduler.Fatal($"The prepared buffer allocation range is invalid: address=0x{range.Address:X16} size=0x{range.Size:X16}."); + pages[index] = new GuestSpan(first * CachingPageSize, (end - first) * CachingPageSize); + } + pages.Sort(static (left, right) => left.Address.CompareTo(right.Address)); + + for (var index = 0; index < pages.Length;) + { + var range = pages[index]; + var next = index + 1; + // AcquireVertexBuffers also coalesces touching byte ranges. Reserve their + // whole union now, so its later acquisition cannot replace a shader binding. + while (next < pages.Length && pages[next].Address <= range.End) + { + range = new GuestSpan(range.Address, Math.Max(range.End, pages[next].End) - range.Address); + next++; + } + + // Disjoint read-only geometry can still use the stream ring, without another cache allocation. + if (next > index + 1 || _registry.HasOverlap(range.Address, range.Size)) + _ = FindBuffer(range.Address, range.Size); + index = next; + } + } + + // SHARPEMU_TRACE_COHERENCE=1 (diagnostic): after a presented frame, with nothing recorded and + // the last submitted GPU work complete, a page the tracker calls clean (no CPU-dirty, GPU-dirty + // or GPU-modified bytes, no cached image over it) must hold the same bytes in guest memory and + // in the mapped buffer. Each check compares the next slice of mapped buffers and reports any + // page that differs, rechecking afterwards that a concurrent guest store did not dirty it. + private static readonly bool TraceCoherence = Environment.GetEnvironmentVariable("SHARPEMU_TRACE_COHERENCE") == "1"; + private static readonly ulong CoherenceBytesPerCheck = + (ulong.TryParse(Environment.GetEnvironmentVariable("SHARPEMU_TRACE_COHERENCE_MB"), out var coherenceMb) && coherenceMb != 0 ? coherenceMb : 8) * 1024 * 1024; + private long _coherenceFrames; + private int _coherenceBufferCursor; + private ulong _coherencePageCursor; + private long _coherencePagesChecked; + private long _coherenceDivergences; + private long _coherenceChecks; + + public void CheckCoherence() + { + if (!TraceCoherence || (++_coherenceFrames & 3) != 0) + { + return; + } + + // Drain everything recorded and run the completion work, so mapped memory holds the GPU's last word. + if (_scheduler.Active) + { + _scheduler.FinishMemoryAccess(); + } + else if (_scheduler.HasUnsubmittedCommands) + { + return; + } + + var lastSubmitted = _scheduler.CurrentTick - 1; + if (lastSubmitted == 0) + { + return; + } + + _scheduler.WaitForSubmittedTick(lastSubmitted); + var identifiers = _registry.SnapshotRegisteredIdentifiers(); + if (identifiers.Length == 0) + { + return; + } + + var images = ImageCache; + Span guest = stackalloc byte[(int)TrackerLayout.PageBytes]; + var budget = CoherenceBytesPerCheck; + for (var visited = 0; visited < identifiers.Length && budget != 0; visited++) + { + _coherenceBufferCursor %= identifiers.Length; + var buffer = _registry.TryGetRegisteredBuffer(identifiers[_coherenceBufferCursor]); + if (buffer is null || buffer.MappedPointer == null || buffer.Size < TrackerLayout.PageBytes) + { + _coherenceBufferCursor++; + _coherencePageCursor = 0; + continue; + } + + var firstPage = (buffer.CpuAddress + TrackerLayout.PageBytes - 1) & ~(TrackerLayout.PageBytes - 1); + var endPage = (buffer.CpuAddress + buffer.Size) & ~(TrackerLayout.PageBytes - 1); + var page = Math.Max(firstPage, _coherencePageCursor); + for (; page < endPage && budget != 0; page += TrackerLayout.PageBytes, budget -= TrackerLayout.PageBytes) + { + if (!IsCoherenceCandidate(page, images) || !_backing.TryReadBacking(page, guest)) + { + continue; + } + + _coherencePagesChecked++; + var gpu = new ReadOnlySpan(buffer.MappedPointer + buffer.Offset(page), (int)TrackerLayout.PageBytes); + var diff = guest.CommonPrefixLength(gpu); + if (diff == guest.Length || !IsCoherenceCandidate(page, images)) + { + continue; + } + + var differing = 0; + for (var index = diff; index < guest.Length; index++) + { + differing += guest[index] != gpu[index] ? 1 : 0; + } + + var divergences = ++_coherenceDivergences; + GuestGpuMemoryHook.SelectDivergenceTracePage(page); + if (divergences <= 40 || divergences % 100 == 0) + { + var from = diff & ~15; + var length = Math.Min(32, guest.Length - from); + var hot = _tracker.IsCpuWriteHotRange(page, TrackerLayout.PageBytes); + var guestHex = Convert.ToHexString(guest.Slice(from, length)); + var gpuHex = Convert.ToHexString(gpu.Slice(from, length)); + Console.Error.WriteLine(FormattableString.Invariant( + $"[GPU][COHERENCE] time={DateTime.Now:HH:mm:ss} page=0x{page:X} buffer=0x{buffer.CpuAddress:X}+0x{buffer.Size:X} first_diff=+0x{diff:X} differing_bytes={differing} hot={hot} last_gpu_write={buffer.LastGpuWriteTick} completed={_scheduler.Timeline.CompletedTick} guest={guestHex} gpu={gpuHex} n={divergences}")); + } + } + + if (page >= endPage) + { + _coherenceBufferCursor++; + _coherencePageCursor = 0; + } + else + { + _coherencePageCursor = page; + } + } + + if ((++_coherenceChecks % 64) == 0) + { + Console.Error.WriteLine(FormattableString.Invariant( + $"[GPU][COHERENCE] time={DateTime.Now:HH:mm:ss} summary pages_checked={_coherencePagesChecked} divergences={_coherenceDivergences} buffers={identifiers.Length}")); + } + } + + private bool IsCoherenceCandidate(ulong page, IGuestImageCache? images) => + images is not null && + _tracker.HasRegion(page, TrackerLayout.PageBytes) && + !_tracker.HasCpuDirtyPages(page, TrackerLayout.PageBytes) && + !_tracker.HasGpuDirtyPages(page, TrackerLayout.PageBytes) && + !_gpuModifiedRanges.Overlaps(page, TrackerLayout.PageBytes) && + !images.QueryRegion(page, TrackerLayout.PageBytes).ImageBytes; + + public (GpuBuffer Buffer, ulong Offset) ObtainBuffer(ulong guestAddress, ulong size, bool isWritten, bool isTexelBuffer = false, ResourceSlotIdentifier bufferIdentifier = default, + bool requiresDeviceAddress = false) { using var profileScope = RenderPhaseProfile.MeasureDetail(RenderPhaseProfile.Phase.BufferAcquisitionChecks); var command = _scheduler.Current; @@ -207,7 +375,14 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable throw SubmissionScheduler.Fatal("A buffer request requires a command buffer that is recording."); } - if (!isWritten && + if (isWritten && isTexelBuffer) + { + // Preserving a partial image write can merge this buffer with the image's whole + // backing range. Finish that before returning a handle the shader will write. + RequireImageCache().InvalidateMemoryFromGpu(guestAddress, size); + } + + if (!requiresDeviceAddress && !isWritten && !_tracker.HasGpuDirtyPages(guestAddress, size) && _tracker.HasCpuDirtyPages(guestAddress, size) && (size <= CachingPageSize || (!isTexelBuffer && _tracker.IsCpuWriteHotRange(guestAddress, size)))) @@ -253,11 +428,63 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable } NoteHotWindowWrite(guestAddress, size); + WatchEvent(guestAddress, size, "writable-binding"); } return (buffer, buffer.Offset(guestAddress)); } + // A texture streamed into its address range maps only its resident mips: the mip tail + // and small levels sit at the start of the chain and level 0 at its end. The GPU samples + // only resident levels, so the unmapped pages of a whole-chain upload read as zero. + private bool TryReadResidentImagePages(ulong guestAddress, Span destination) + { + const ulong page = 1UL << 12; + var size = (ulong)destination.Length; + if (size == 0 || !_backing.IsBackedRange(guestAddress, 1)) + { + return false; + } + + // Only a resident prefix qualifies: a hole between backed pages is not a mip chain. + var resident = true; + for (ulong offset = 0; offset < size;) + { + var chunk = Math.Min(page - ((guestAddress + offset) & (page - 1)), size - offset); + var target = destination.Slice((int)offset, (int)chunk); + var backed = _backing.IsBackedRange(guestAddress + offset, chunk); + if (backed && !resident) + { + return false; + } + + resident = backed && _backing.TryReadBacking(guestAddress + offset, target); + if (!resident) + { + target.Clear(); + } + + offset += chunk; + } + + return true; + } + + // The image cache publishes an image's GPU-only contents into the buffer over the same bytes + // before it drops the image. The buffer then owns them, as after any GPU write. + public GpuBuffer? ObtainBufferForImageWriteBack(ulong guestAddress, ulong size) + { + if (!IsValidRange(guestAddress, size) || !_backing.IsBackedRange(guestAddress, size)) + { + return null; + } + + return ObtainBuffer(guestAddress, size, isWritten: true).Buffer; + } + + public bool IsGuestRangeMapped(ulong guestAddress, ulong size) => + IsValidRange(guestAddress, size) && _backing.IsBackedRange(guestAddress, size); + public (GpuBuffer Buffer, ulong Offset) ObtainBufferForImage(ulong guestAddress, ulong size) { if (!IsValidRange(guestAddress, size)) @@ -281,9 +508,17 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable owner = _registry.GetBuffer(FindBuffer(guestAddress, size)); } - if (owner != null && !cpuModified && (!gpuModified || hasDirtyBufferSource)) + if (owner != null && (!gpuModified || hasDirtyBufferSource)) { TouchBuffer(owner); + if (cpuModified) + { + // Tracking clears complete pages, even for a subpage image. The regular + // uploader stages those complete runs; staging only the image would copy + // unprepared bytes over neighbouring textures and mark them clean. + _ = SynchronizeBuffer(owner, guestAddress, size, isWritten: false, isTexelBuffer: true, + preserveCpuWriteHotPages: false, readImageBacking: true); + } return (owner, owner.Offset(guestAddress)); } @@ -299,7 +534,8 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable } if (!_backing.TryReadBacking(guestAddress, _staging.Mapped.Slice((int)stageOffset, (int)size)) && !KernelMemoryCompatExports.TryReadPrtBacking(_backing, guestAddress, - _staging.Mapped.Slice((int)stageOffset, (int)size))) + _staging.Mapped.Slice((int)stageOffset, (int)size)) && + !TryReadResidentImagePages(guestAddress, _staging.Mapped.Slice((int)stageOffset, (int)size))) { throw SubmissionScheduler.Fatal( $"Could not read the mapped guest image backing: address=0x{guestAddress:X16} size=0x{size:X16} range_backed={_backing.IsBackedRange(guestAddress, size)} first_byte_backed={_backing.IsBackedRange(guestAddress, 1)} last_byte_backed={_backing.IsBackedRange(guestAddress + size - 1, 1)} tick={_scheduler.CurrentTick}."); @@ -319,15 +555,35 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable } TouchBuffer(owner); - // Tracking reports whole dirty pages, which can extend beyond the image's - // staging slice. Upload those pages through the normal buffer uploader so - // every byte marked clean is copied from its actual guest backing. - _ = SynchronizeBuffer(owner, guestAddress, size, false, false); + _ = SynchronizeBuffer(owner, guestAddress, size, isWritten: false, isTexelBuffer: true, + preserveCpuWriteHotPages: false, readImageBacking: true); return (owner, owner.Offset(guestAddress)); } + private GuestBufferSourceReader? _tryReadImageSource; + + // Image acquisition already holds the image-cache lock. Read CPU-owned backing + // directly instead of faulting through guest memory and reentering that cache. + private bool TryReadImageSource(ulong address, Span destination) => + _backing.TryReadBacking(address, destination) || + KernelMemoryCompatExports.TryReadPrtBacking(_backing, address, destination) || + TryReadResidentImagePages(address, destination); + + // The command worker can read the stable backing alias without changing CPU page + // permissions. A GPU counter elsewhere on the same page is not a dependency of + // the packet bytes being decoded. Actual GPU-written bytes retain the slow path. + private static readonly bool CommandBackingReadsEnabled = + Environment.GetEnvironmentVariable("SHARPEMU_COMMAND_BACKING_READS") != "0"; + + public bool TryReadCommandBacking(ulong address, Span destination) => + CommandBackingReadsEnabled && destination.Length != 0 && + !_gpuModifiedRanges.Overlaps(address, (ulong)destination.Length) && + ImageCache is { } images && !images.HasGpuModifiedImageBytes(address, (ulong)destination.Length) && + _backing.TryReadBacking(address, destination); + public void WriteHostMemory(ulong guestAddress, ReadOnlySpan data) { + WatchEvent(guestAddress, (ulong)data.Length, "host-dma-write"); if (guestAddress == 0 || data.IsEmpty || (ulong)data.Length > ulong.MaxValue - guestAddress) { throw SubmissionScheduler.Fatal("The host DMA write range is invalid."); @@ -340,6 +596,27 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable // Registered buffers are ordered and disjoint: walk only the ones the write overlaps. var end = guestAddress + (ulong)data.Length; + if (DeferHostWrites) + { + // When no GPU-written byte shares the written pages, the pages can simply turn + // CPU-dirty: every later GPU reader synchronizes CPU-dirty pages first (bound buffers + // on obtain, device-address programs at the next visibility point), so the bytes + // arrive in command order without a staging copy that ends the rendering scope. + var firstPage = guestAddress & ~(TrackerLayout.PageBytes - 1); + var pagesEnd = (end + TrackerLayout.PageBytes - 1) & ~(TrackerLayout.PageBytes - 1); + if (!_gpuModifiedRanges.Overlaps(firstPage, pagesEnd - firstPage) && + !_tracker.HasGpuDirtyPages(firstPage, pagesEnd - firstPage)) + { + if (IsRegionRegistered(guestAddress, (ulong)data.Length)) + { + _tracker.MarkCpuDirtyPages(guestAddress, (ulong)data.Length); + NoteMemoryVisibilityPoint(); + } + + return; + } + } + for (var index = _registry.FindFirstOverlappingIndex(guestAddress); index < _registry.RegisteredCount && _registry.GetRegisteredAddress(index) < end; index++) @@ -359,8 +636,12 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable } } + private static readonly bool DeferHostWrites = + Environment.GetEnvironmentVariable("SHARPEMU_DEFER_HOST_WRITES") != "0"; + public void FillBuffer(ulong guestAddress, ulong size, uint value, bool isGds) { + WatchEvent(guestAddress, size, "fill"); if ((guestAddress & 3) != 0 || size == 0 || (size & 3) != 0 || size > ulong.MaxValue - guestAddress) { throw SubmissionScheduler.Fatal("The fill range must be aligned to four bytes."); @@ -441,11 +722,15 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable } } + // The fill pattern staged per thread; DCC fills arrive with every compressed target clear. + [ThreadStatic] + private static uint[]? _fillPattern; + private bool TryWriteFillToBacking(ulong guestAddress, ulong size, uint value) { - var values = new uint[(int)Math.Min(size / sizeof(uint), 4096)]; - Array.Fill(values, value); - var bytes = MemoryMarshal.AsBytes(values); + var values = (_fillPattern ??= new uint[4096]).AsSpan(0, (int)Math.Min(size / sizeof(uint), 4096)); + values.Fill(value); + var bytes = MemoryMarshal.AsBytes(values); for (ulong offset = 0; offset < size;) { var chunk = (int)Math.Min(size - offset, (ulong)bytes.Length); @@ -464,6 +749,7 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable public bool TryCopyWordsOnHost(ulong destination, ulong source, ulong sourceWords, ulong words) { + if (WatchedRanges.Length != 0) WatchEvent(destination, words * 4, $"host-copy src=0x{source:X}"); if (destination == 0 || source == 0 || sourceWords == 0 || words == 0 || ((destination | source) & 3) != 0 || words > MaxHostCopyWords || sourceWords > MaxHostCopyWords) { @@ -528,6 +814,7 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable public void CopyBuffer(ulong dstVaddr, ulong srcVaddr, ulong size, bool dstGds, bool srcGds) { + if (WatchedRanges.Length != 0) WatchEvent(dstVaddr, size, $"dma-copy src=0x{srcVaddr:X}"); var dstMemory = !dstGds; var srcMemory = !srcGds; if ((dstMemory && dstVaddr == 0) || (srcMemory && srcVaddr == 0) || size == 0 || @@ -613,6 +900,197 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable public void ProcessFaultBuffer() => _faults.ProcessFaultBuffer(); + private static long _reportedDeviceWriteConflicts; + + // Pages that shaders wrote through device addresses (runtime V#s, global and FLAT stores, + // atomics) are reported by the GPU after the fact; no binding announced them. They become + // GPU-owned like any written binding, so CPU reads download them and a CPU write to the page + // flushes them first instead of a later upload overwriting them with stale guest bytes. + internal void NoteDeviceAddressWrites(ulong guestAddress, ulong size) + { + if (!IsValidRange(guestAddress, size)) + { + return; + } + + var end = guestAddress + size; + var marked = false; + for (var page = guestAddress & ~(TrackerLayout.PageBytes - 1); page < end; page += TrackerLayout.PageBytes) + { + if (FindOwner(page, TrackerLayout.PageBytes) is not { } owner) + { + continue; + } + + var conflicts = _tracker.MarkGpuWrittenPages(page, TrackerLayout.PageBytes); + WatchEvent(page, TrackerLayout.PageBytes, conflicts != 0 ? "device-write-on-cpu-dirty-page" : "device-write"); + if (conflicts != 0) + { + // The CPU dirtied the page before the write was reported: the page stays + // CPU-owned and the shader's bytes are merged into guest memory instead. + if (_sharedPages.Contains(page)) + { + marked |= PullSharedPage(page, owner); + } + else + { + AdoptLateSharedPage(page, owner); + } + + continue; + } + + // The first CPU access downloads the page; the shadow starts from the downloaded bytes. + if (!_sharedPages.Contains(page)) + { + lock (_shaderWrittenPages) + { + if (_shaderWrittenPages.Count < MaxShaderWrittenPages) + { + _shaderWrittenPages[page] = owner; + Volatile.Write(ref _shaderWrittenPageCount, _shaderWrittenPages.Count); + } + } + } + + if (!_gpuModifiedRanges.Contains(page, TrackerLayout.PageBytes)) + { + _gpuModifiedRanges.Add(page, TrackerLayout.PageBytes); + Interlocked.Increment(ref _gpuModifiedVersion); + } + + owner.NoteGpuWrite(); + marked = true; + if (GuestGpuMemoryHook.Traces(page, TrackerLayout.PageBytes)) + GuestGpuMemoryHook.Trace(page, TrackerLayout.PageBytes, $"device-address-write-reported submission_tick={_scheduler.CurrentTick}"); + } + + if (marked) + { + RequireImageCache().InvalidateMemoryCopiesFromGpu(guestAddress, size); + } + } + + // Pages shaders write through device addresses; see SharedPageShadows. 64 MiB of shadows at most. + private const int SharedPageCapacity = 16384; + private readonly SharedPageShadows _sharedPages = new(TrackerLayout.PageBytes, SharedPageCapacity); + private static long _reportedLateSharedPages; + private static long _reportedSharedPageMerges; + private static int _reportedSharedPageCapacity; + + // Pages shaders wrote that have no shadow yet, with their owner at the time of the write. + private const int MaxShaderWrittenPages = 1 << 20; + private readonly Dictionary _shaderWrittenPages = new(); + private int _shaderWrittenPageCount; + + // A page the CPU dirtied before its first shader write was reported: shadowed from the GPU copy. + private void AdoptLateSharedPage(ulong page, GpuBuffer owner) + { + var adopted = owner.MappedPointer != null && owner.IsCoherent && + _sharedPages.Adopt(page, owner, new ReadOnlySpan(owner.MappedPointer + owner.Offset(page), (int)TrackerLayout.PageBytes), + _scheduler.CurrentTick); + if (!adopted && _sharedPages.Count >= SharedPageCapacity && Interlocked.Exchange(ref _reportedSharedPageCapacity, 1) == 0) + { + Console.Error.WriteLine("[GPU][WARN] The shared page shadows are full; CPU writes to further shader-written pages may lose shader bytes."); + } + + var reported = Interlocked.Increment(ref adopted ? ref _reportedLateSharedPages : ref _reportedDeviceWriteConflicts); + if (reported <= 16 || (reported & (reported - 1)) == 0) + { + Console.Error.WriteLine(adopted + ? FormattableString.Invariant( + $"[GPU][SHARED_PAGE] The CPU dirtied page=0x{page:X16} before its first shader write was reported; shader bytes an upload already overwrote are lost once shared={_sharedPages.Count} n={reported}") + : FormattableString.Invariant( + $"[GPU][WARN] A device-address write reached a page the CPU dirtied and the page has no shadow: page=0x{page:X16} count={reported}")); + } + } + + // Writes downloaded GPU bytes into guest memory; shadows of shared pages take them too. + private bool WriteDownloaded(ulong address, ReadOnlySpan bytes) + { + if (!_backing.TryWriteBacking(address, bytes)) + { + return false; + } + + _sharedPages.NoteDownloaded(address, bytes); + AdoptDownloadedPages(address, (ulong)bytes.Length); + return true; + } + + // Downloaded shader-written pages now hold in guest memory what the GPU copy holds (the bytes + // outside the download were not GPU-written, so they already agreed): shadow them from there. + private void AdoptDownloadedPages(ulong address, ulong size) + { + if (Volatile.Read(ref _shaderWrittenPageCount) == 0) + { + return; + } + + Span guest = stackalloc byte[(int)TrackerLayout.PageBytes]; + var end = address + size; + for (var page = address & ~(TrackerLayout.PageBytes - 1); page < end; page += TrackerLayout.PageBytes) + { + GpuBuffer? owner; + lock (_shaderWrittenPages) + { + if (!_shaderWrittenPages.Remove(page, out owner)) + { + continue; + } + + Volatile.Write(ref _shaderWrittenPageCount, _shaderWrittenPages.Count); + } + + if (_backing.TryReadBacking(page, guest) && !_sharedPages.Adopt(page, owner!, guest) && + Interlocked.Exchange(ref _reportedSharedPageCapacity, 1) == 0) + { + Console.Error.WriteLine("[GPU][WARN] The shared page shadows are full; CPU writes to further shader-written pages may lose shader bytes."); + } + } + } + + // Merges what shaders wrote on a shared page into guest memory; true when bytes were merged. + private bool PullSharedPage(ulong page, GpuBuffer owner) + { + if (_sharedPages.Count == 0 || owner.MappedPointer == null || !owner.IsCoherent || !_sharedPages.Contains(page)) + { + return false; + } + + // The GPU may still write the page: merge one snapshot of it. + Span gpu = stackalloc byte[(int)TrackerLayout.PageBytes]; + new ReadOnlySpan(owner.MappedPointer + owner.Offset(page), gpu.Length).CopyTo(gpu); + var guest = new byte[TrackerLayout.PageBytes]; + if (!_backing.TryReadBacking(page, guest)) + { + return false; + } + + var count = _sharedPages.PullGpuWrites(page, owner, gpu, guest, _scheduler.Timeline.CompletedTick, (offset, length) => + { + if (!_backing.TryWriteBacking(page + (ulong)offset, guest.AsSpan(offset, length))) + { + throw SubmissionScheduler.Fatal($"Could not merge shader writes into guest memory: addr=0x{page + (ulong)offset:X16} size=0x{length:X}"); + } + }); + if (count == 0) + { + return false; + } + + var reported = Interlocked.Increment(ref _reportedSharedPageMerges); + if (reported <= 16 || (reported & (reported - 1)) == 0) + { + Console.Error.WriteLine(FormattableString.Invariant( + $"[GPU][SHARED_PAGE] merged shader bytes into a CPU-dirty page=0x{page:X16} bytes={count} shared={_sharedPages.Count} merged_bytes={_sharedPages.MergedBytes} skipped_upload_bytes={_sharedPages.SkippedUploadBytes} n={reported}")); + } + + if (GuestGpuMemoryHook.Traces(page, TrackerLayout.PageBytes)) + GuestGpuMemoryHook.Trace(page, TrackerLayout.PageBytes, $"shared-page-merged bytes={count} submission_tick={_scheduler.CurrentTick}"); + return true; + } + // Uploads every mapped range before a BDA draw; the fault pass runs at the next collection. // Every device-address program prepares all GPU-mapped memory; visiting each registered // buffer per dispatch cost ~17 % of the Demon's Souls render thread. The same work is done @@ -623,6 +1101,58 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable private ulong _bdaTouchTick = ulong.MaxValue; private ulong _bdaTouchMapping; + // Ranges shaders reached through device addresses outside the GPU mappings: pointers + // into ordinary guest memory, which the GPU reads too. Only a fault reveals them, so + // they are kept and touched like the mappings. Otherwise their buffers age out while + // still in use, fault again, and the cache collects and re-creates them every frame. + private readonly SpanSet _deviceAddressFaultSpans = new(); + + // Fault ranges inside a known guest mapping; they are forgotten once it is unmapped. + private readonly SpanSet _mappedDeviceAddressFaultSpans = new(); + + internal void NoteDeviceAddressFault(ulong guestAddress, ulong size, bool insideGuestMapping = false) + { + _deviceAddressFaultSpans.Add(guestAddress, size); + if (insideGuestMapping) + _mappedDeviceAddressFaultSpans.Add(guestAddress, size); + } + + // A guest buffer the GPU reads through pointers (a per-frame ring, for one) is reached a + // page at a time, and every first touch reads zeros for that frame. A fault therefore + // brings in the aligned window around it, clipped to the guest mapping that holds it. + internal const ulong DeviceAddressFaultWindow = 2UL << 20; + + internal static GuestSpan DeviceAddressFaultSpan(ulong pageAddress, ulong pageSize, ulong mappingStart, ulong mappingLength) + { + if (mappingLength == 0 || pageAddress < mappingStart || pageAddress - mappingStart >= mappingLength) + { + return new GuestSpan(pageAddress, pageSize); + } + + var mappingEnd = mappingStart + mappingLength; + var windowStart = Math.Max(pageAddress & ~(DeviceAddressFaultWindow - 1), mappingStart); + var windowEnd = Math.Min((pageAddress & ~(DeviceAddressFaultWindow - 1)) + DeviceAddressFaultWindow, mappingEnd); + return new GuestSpan(windowStart, windowEnd - windowStart); + } + + // Forgets fault ranges whose guest mapping is gone, so their buffers can age out again. + private void PruneUnmappedDeviceAddressFaults() + { + List? unmapped = null; + _mappedDeviceAddressFaultSpans.ForEach((start, size) => + { + if (!KernelMemoryCompatExports.TryGetMappedRange(start, out var mappingStart, out var mappingLength) || + start + size > mappingStart + mappingLength) + (unmapped ??= []).Add(new GuestSpan(start, size)); + }); + if (unmapped is null) return; + foreach (var span in unmapped) + { + _mappedDeviceAddressFaultSpans.Remove(span.Address, span.Size); + _deviceAddressFaultSpans.Remove(span.Address, span.Size); + } + } + public static ulong MappingKey(IReadOnlyCollection spans) { var mapping = (ulong)spans.Count; @@ -657,17 +1187,34 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable { foreach (var span in spans) TouchBuffersInRange(span.Address, span.Size); + PruneUnmappedDeviceAddressFaults(); + _deviceAddressFaultSpans.ForEach(_touchBuffersInRange ??= TouchBuffersInRange); _bdaTouchTick = _retirementPolicy.CurrentTick; _bdaTouchMapping = mapping; } - var epoch = _tracker.CpuDirtyEpoch; - if (epoch != _bdaSweepEpoch || mapping != _bdaSweepMapping) + // Device-address programs may read any mapped page, so CPU writes are swept into the + // buffers - but only at memory visibility points (a submission slice starting, a wait + // packet satisfied). A CPU write after the submission began carries no ordering + // guarantee for its commands, and sweeping on every draw would end the render pass + // for every page the guest streams while the stream is being translated. + // Hot pages remain writable, so later CPU writes need not advance CpuDirtyEpoch. + // Every visibility point must revisit their dirty ranges even with an unchanged epoch. + if (mapping != _bdaSweepMapping || _bdaVisibilityPending || SweepBdaOnEveryDraw) { - foreach (var span in spans) - _tracker.ForEachPossiblyCpuDirtyRange(span.Address, span.Size, _uploadDirtyBuffersInRange ??= UploadDirtyBuffersInRange); - _bdaSweepEpoch = epoch; + BeginUploadBatch(); + try + { + foreach (var span in spans) + _tracker.ForEachPossiblyCpuDirtyRange(span.Address, span.Size, _uploadDirtyBuffersInRange ??= UploadDirtyBuffersInRange); + } + finally + { + EndUploadBatch(); + } + _bdaSweepMapping = mapping; + _bdaVisibilityPending = false; } } @@ -678,9 +1225,127 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable } private Action? _uploadDirtyBuffersInRange; - private long _bdaSweepEpoch = -1; + private Action? _touchBuffersInRange; + + private static readonly bool PreserveHotPagesInSweeps = + Environment.GetEnvironmentVariable("SHARPEMU_BDA_SWEEP_REPROTECT_HOT") != "1"; + private bool _bdaVisibilityPending = true; + + private static readonly bool SweepBdaOnEveryDraw = + Environment.GetEnvironmentVariable("SHARPEMU_BDA_SWEEP_EVERY_DRAW") == "1"; + + // Called once per presented frame; see GuestPageTracker.DecayCpuWriteHeat. + public void DecayCpuWriteHeat() + { + if (!KeepCpuWriteHeatAcrossFrames) + { + _tracker.DecayCpuWriteHeat(); + } + } + + private static readonly bool KeepCpuWriteHeatAcrossFrames = + Environment.GetEnvironmentVariable("SHARPEMU_KEEP_CPU_WRITE_HEAT") == "1"; + + public void NoteMemoryVisibilityPoint() + { + _bdaVisibilityPending = true; + _visibilityGeneration++; + _reportedVisibilityPoints++; + } + + // A command-processor write (labels, EOP and WRITE_DATA payloads) is ordered in the stream: + // only buffers holding the written bytes must copy their hot pages again. Advancing the + // generation here instead would restart every buffer thousands of times a second. + public void NoteCommandProcessorWrite(ulong guestAddress, ulong size) + { + _bdaVisibilityPending = true; + _reportedProcessorWrites++; + var end = guestAddress + size; + var index = _registry.FindFirstOverlappingIndex(guestAddress); + for (; index < _registry.RegisteredCount && _registry.GetRegisteredAddress(index) < end; index++) + { + _registry.GetBuffer(_registry.GetRegisteredIdentifier(index)).HotSyncGeneration = 0; + } + } + + private static long _reportedProcessorWrites; + private ulong _bdaSweepMapping; + // Advances at every memory visibility point; starts above a new buffer's zero generation. + private long _visibilityGeneration = 1; + + private static readonly bool ResyncHotPagesOnEveryDraw = + Environment.GetEnvironmentVariable("SHARPEMU_BDA_RESYNC_HOT_EVERY_DRAW") == "1"; + + private static long _reportedVisibilityPoints; + private static long _reportedHotSyncFull; + private static long _reportedHotSyncSkipped; + + public static string TakeDeviceAddressSyncReport() => FormattableString.Invariant( + $"[PERF][BDA_SYNC] visibility_points={Interlocked.Exchange(ref _reportedVisibilityPoints, 0)} processor_writes={Interlocked.Exchange(ref _reportedProcessorWrites, 0)} full_syncs={Interlocked.Exchange(ref _reportedHotSyncFull, 0)} hot_skipped_syncs={Interlocked.Exchange(ref _reportedHotSyncSkipped, 0)}"); + + // A draw's read-only device-address range. CPU-write-hot pages stay writable, so they + // read as dirty on every draw; copying them again for each draw of one visibility + // generation re-reads bytes the guest had to finish writing before that point (the + // same ordering rule the PrepareBda sweep relies on). Each buffer remembers the range + // it copied hot pages for in the current generation and skips them inside it; pages + // that fault dirty are still copied on every call. + public void SynchronizeDeviceAddressRange(ulong guestAddress, ulong size) + { + if (ResyncHotPagesOnEveryDraw || GuestGpuMemoryHook.TraceAddress != 0) + { + SynchronizeBuffersInRange(guestAddress, size); + return; + } + + var end = guestAddress + size; + var generation = _visibilityGeneration; + var index = _registry.FindFirstOverlappingIndex(guestAddress); + for (; index < _registry.RegisteredCount && _registry.GetRegisteredAddress(index) < end; index++) + { + var identifier = _registry.GetRegisteredIdentifier(index); + var buffer = _registry.GetBuffer(identifier); + var start = Math.Max(buffer.CpuAddress, guestAddress); + var finish = Math.Min(buffer.CpuAddress + buffer.Size, end); + if (start >= finish) + { + continue; + } + + TouchBuffer(identifier); + if (!_tracker.HasCpuDirtyPages(start, finish - start)) + { + continue; + } + + var covered = buffer.HotSyncGeneration == generation && start >= buffer.HotSyncStart && finish <= buffer.HotSyncEnd; + if (covered) + { + _reportedHotSyncSkipped++; + } + else + { + _reportedHotSyncFull++; + // Grow the remembered range when the new one touches it; otherwise restart it. + if (buffer.HotSyncGeneration == generation && start <= buffer.HotSyncEnd && finish >= buffer.HotSyncStart) + { + buffer.HotSyncStart = Math.Min(buffer.HotSyncStart, start); + buffer.HotSyncEnd = Math.Max(buffer.HotSyncEnd, finish); + } + else + { + buffer.HotSyncGeneration = generation; + buffer.HotSyncStart = start; + buffer.HotSyncEnd = finish; + } + } + + _ = SynchronizeBuffer(buffer, start, finish - start, false, false, + preserveCpuWriteHotPages: PreserveHotPagesInSweeps, skipCpuWriteHotPages: covered); + } + } + private void TouchBuffersInRange(ulong guestAddress, ulong size) { var end = guestAddress + size; @@ -699,8 +1364,10 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable var buffer = _registry.GetBuffer(_registry.GetRegisteredIdentifier(index)); var start = Math.Max(buffer.CpuAddress, guestAddress); var finish = Math.Min(buffer.CpuAddress + buffer.Size, end); + // Hot pages stay writable and are copied again on the next sweep: a few hundred KiB of + // ring memory per sweep costs far less than a protection fault per page per frame. if (start < finish && _tracker.MayHaveCpuDirtyPages(start, finish - start)) - _ = SynchronizeBuffer(buffer, start, finish - start, false, false, preserveCpuWriteHotPages: false); + _ = SynchronizeBuffer(buffer, start, finish - start, false, false, preserveCpuWriteHotPages: PreserveHotPagesInSweeps); } } @@ -726,7 +1393,7 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable // SynchronizeBuffer takes for this read-only call), and the block summary // answers that without a lock for the common all-clean case. if (_tracker.HasCpuDirtyPages(start, finish - start)) - _ = SynchronizeBuffer(buffer, start, finish - start, false, false, preserveCpuWriteHotPages: false); + _ = SynchronizeBuffer(buffer, start, finish - start, false, false, preserveCpuWriteHotPages: PreserveHotPagesInSweeps); } } } @@ -751,11 +1418,12 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable { using var foreignRead = _device.Slabs.BeginForeignRead(); ProcessPendingFaultBuffer(); - if (!_retirementPolicy.TryBeginCollection(_registry.RegisteredBytes, out var retirement)) + if (!_retirementPolicy.TryBeginCollection(TotalUsedMemory, out var retirement)) { return; } + var dirtyBuffers = new List(); var copies = new List(); var retireCount = 0; @@ -896,24 +1564,45 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable $"unsupported buffer readback from an asynchronous GPU completion, addr=0x{guestAddress:X16} size=0x{size:X16}"); } - if (!_relay.IsGpuQueueThread && AsyncReadback is { } readback && GuestReadsAwaitOffQueue) + if (!_relay.IsGpuQueueThread && GuestReadsAwaitOffQueue && (AsyncReadback is not null || MainQueuePendingReads)) { - PendingDownload? pending = null; - if (_relay.TryRunOnGpuQueue(() => pending = BeginReadMemoryOnGpu(guestAddress, size, isWrite, source, allowPending: true))) + // A GPU write that lands after the copy was recorded makes the read stale. Retrying + // synchronously stalls the command worker until the GPU drains, so re-issue the read and + // wait here again; only the last attempt falls back to the synchronous read. + for (var attempt = 1; ; attempt++) { + PendingDownload? pending = null; + if (!_relay.TryRunOnGpuQueue(() => pending = BeginReadMemoryOnGpu(guestAddress, size, isWrite, source, allowPending: true))) + { + return AwaitShutdown(); + } + if (pending is null) { return true; } - readback.Wait(pending.Ticket); - if (_relay.TryRunOnGpuQueue(() => CompleteReadMemoryOnGpu(pending, retrySynchronously: true))) + if (pending.Ticket is { } ticket) + { + (AsyncReadback ?? throw SubmissionScheduler.Fatal("The pending readback lost its queue.")).Wait(ticket); + } + else + { + _scheduler.WaitForSubmittedTick(pending.MainQueueTick); + } + + var finalAttempt = attempt >= PendingReadAttempts; + var applied = false; + if (!_relay.TryRunOnGpuQueue(() => applied = CompleteReadMemoryOnGpu(pending, retrySynchronously: finalAttempt))) + { + return AwaitShutdown(); + } + + if (applied || finalAttempt) { return true; } } - - return AwaitShutdown(); } var onQueue = _relay.IsGpuQueueThread; @@ -932,11 +1621,23 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable } private const ulong ReadbackWindowBytes = 512 * 1024; + // Uncached reads of device-local memory run near 300 MB/s, and most mapped reads want one + // descriptor word or packet dword: one tracker page is the smallest window that can turn clean. + // SHARPEMU_MAPPED_READBACK_KB overrides it (a power of two) for comparisons. + private static readonly ulong MappedReadbackWindowBytes = + ulong.TryParse(Environment.GetEnvironmentVariable("SHARPEMU_MAPPED_READBACK_KB"), out var mappedKb) && mappedKb != 0 && + (mappedKb & (mappedKb - 1)) == 0 ? mappedKb * 1024 : TrackerLayout.PageBytes; private const long HotWindowLifetime = 512; private const int MaxEagerReadbacks = 4; private readonly Dictionary _hotWindows = new(); private readonly HashSet _eagerCandidates = new(); private readonly List _eagerDownloads = new(); + // Consecutive eager readbacks of a window the GPU rewrote before the CPU read them. + // A window that keeps wasting its copies stops getting them until the counts reset. + private readonly Dictionary _eagerMisses = new(); + private const int MaxEagerMisses = 2; + private const long EagerMissResetBatches = 256; + private static long _reportedEagerSkipped; private long _batchSerial; private static long _reportedEagerStarted; private static long _reportedEagerUsed; @@ -991,10 +1692,16 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable if (!CompleteReadMemoryOnGpu(pending, retrySynchronously: false)) { Interlocked.Increment(ref _reportedEagerStale); + NoteEagerMiss(pending.GuestAddress); } } } + if (_batchSerial % EagerMissResetBatches == 0) + { + _eagerMisses.Clear(); + } + if ((_batchSerial & 1023) == 0) { foreach (var key in _hotWindows.Where(pair => _batchSerial - pair.Value.LastHit > HotWindowLifetime).Select(pair => pair.Key).ToList()) @@ -1015,6 +1722,12 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable break; } + if (_eagerMisses.TryGetValue(key, out var misses) && misses >= MaxEagerMisses) + { + Interlocked.Increment(ref _reportedEagerSkipped); + continue; + } + if (_hotWindows.TryGetValue(key, out var hot)) { StartEagerReadback(hot); @@ -1086,14 +1799,22 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable if (!CompleteReadMemoryOnGpu(pending, retrySynchronously: false)) { Interlocked.Increment(ref _reportedEagerStale); + NoteEagerMiss(pending.GuestAddress); return false; } + _eagerMisses.Remove(pending.GuestAddress & ~(ReadbackWindowBytes - 1)); EagerReadbacksUsed++; Interlocked.Increment(ref _reportedEagerUsed); return !HasGpuDirtyBytes(guestAddress, size); } + private void NoteEagerMiss(ulong guestAddress) + { + var key = guestAddress & ~(ReadbackWindowBytes - 1); + _eagerMisses[key] = _eagerMisses.TryGetValue(key, out var misses) ? misses + 1 : 1; + } + private void AbandonEagerReadbacks() { if (AsyncReadback is not { } readback) @@ -1142,11 +1863,16 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable private static long _reportedRetried; public static string TakeAsyncReadbackReport() => FormattableString.Invariant( - $"[PERF][ASYNC_READBACK] applied={Interlocked.Exchange(ref _reportedApplied, 0)} retried={Interlocked.Exchange(ref _reportedRetried, 0)} eager_started={Interlocked.Exchange(ref _reportedEagerStarted, 0)} eager_used={Interlocked.Exchange(ref _reportedEagerUsed, 0)} eager_stale={Interlocked.Exchange(ref _reportedEagerStale, 0)}"); + $"[PERF][ASYNC_READBACK] applied={Interlocked.Exchange(ref _reportedApplied, 0)} retried={Interlocked.Exchange(ref _reportedRetried, 0)} eager_started={Interlocked.Exchange(ref _reportedEagerStarted, 0)} eager_used={Interlocked.Exchange(ref _reportedEagerUsed, 0)} eager_stale={Interlocked.Exchange(ref _reportedEagerStale, 0)} eager_skipped={Interlocked.Exchange(ref _reportedEagerSkipped, 0)}"); + // Ticket is set when the readback queue carries the download; otherwise the main queue + // does, into MainQueueBuffer, and MainQueueTick is the submission the guest thread waits for. private sealed record PendingDownload { - public required VulkanAsyncReadback.Ticket Ticket { get; init; } + public VulkanAsyncReadback.Ticket? Ticket { get; init; } + public ulong MainQueueTick { get; init; } + public GpuBuffer? MainQueueBuffer { get; init; } + public ulong[] MainQueueOffsets { get; init; } = []; public required List Copies { get; init; } public required ulong[] WriteTicks { get; init; } public required ulong WindowBegin { get; init; } @@ -1156,6 +1882,9 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable public required bool IsWrite { get; init; } public required GuestMemoryProfile.ReadbackSource Source { get; init; } public required long Started { get; init; } + + // Read from the buffers' own host mappings once the writer tick retires; no copy was recorded. + public bool Mapped { get; init; } } private PendingDownload? BeginReadMemoryOnGpu(ulong guestAddress, ulong size, bool isWrite, GuestMemoryProfile.ReadbackSource source, @@ -1179,6 +1908,56 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable { NoteHotReadback(guestAddress, size); } + if (copies.Count != 0 && _unifiedBuffers && AreMapped(copies, out var writer)) + { + // The mapping is device-local memory the CPU reads uncached across the bus, so a + // mapped read copies only the pages around the request, not the whole window. + copies = CollectReadbackWindow(copies[0].Buffer, guestAddress, size, MappedReadbackWindowBytes, out windowBegin, out windowEnd); + if (writer == 0 || _scheduler.IsTickComplete(writer)) + { + ApplyMappedRead(copies, windowBegin, windowEnd, guestAddress, size, isWrite, readbackStarted, source); + return null; + } + + if (allowPending && writer < _scheduler.CurrentTick) + { + // A guest thread had to wait for this writer: submit later writes to the window right + // after their command, so the next wait ends with them instead of with their whole batch. + NoteHotReadback(guestAddress, size); + + // The writer is already submitted: wait for it alone, not for the work queued after it. + var mappedTicks = new ulong[copies.Count]; + for (var index = 0; index < copies.Count; index++) + { + mappedTicks[index] = copies[index].Buffer.LastGpuWriteTick; + copies[index].Buffer.RetainForeignRead(); + } + + return new PendingDownload + { + Mapped = true, + MainQueueTick = writer, + Copies = copies, + WriteTicks = mappedTicks, + WindowBegin = windowBegin, + WindowEnd = windowEnd, + GuestAddress = guestAddress, + Size = size, + IsWrite = isWrite, + Source = source, + Started = readbackStarted, + }; + } + + if (!allowPending) + { + // On the command worker: submit the writer if it is still recording, then read the mapping. + _scheduler.Wait(writer); + ApplyMappedRead(copies, windowBegin, windowEnd, guestAddress, size, isWrite, readbackStarted, source); + return null; + } + } + if (copies.Count != 0 && allowPending && TryBeginDownloadAsync(copies, out var ticket, out var writeTicks)) { foreach (var copy in copies) @@ -1201,6 +1980,31 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable }; } + if (copies.Count != 0 && allowPending && + TryBeginDownloadOnMainQueue(copies, out var mainTick, out var mainBuffer, out var mainOffsets, out var mainWriteTicks)) + { + foreach (var copy in copies) + { + copy.Buffer.RetainForeignRead(); + } + + return new PendingDownload + { + MainQueueTick = mainTick, + MainQueueBuffer = mainBuffer, + MainQueueOffsets = mainOffsets, + Copies = copies, + WriteTicks = mainWriteTicks, + WindowBegin = windowBegin, + WindowEnd = windowEnd, + GuestAddress = guestAddress, + Size = size, + IsWrite = isWrite, + Source = source, + Started = readbackStarted, + }; + } + if (copies.Count != 0) { DownloadBufferMemory(copies); @@ -1216,9 +2020,75 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable return null; } + // Without a readback queue the main queue copies the pieces into a buffer of their own and + // submits at once; the guest thread then waits for that submission instead of the worker + // waiting for the whole GPU (a guest read of GPU-written memory otherwise idled the worker + // for tens of milliseconds, several times per frame in Silent Hill: The Short Message). + private bool TryBeginDownloadOnMainQueue(List copies, out ulong tick, out GpuBuffer buffer, out ulong[] offsets, out ulong[] writeTicks) + { + tick = 0; + buffer = null!; + offsets = []; + writeTicks = []; + if (!MainQueuePendingReads || copies.Count == 0) + { + return false; + } + + var ticks = new ulong[copies.Count]; + var placements = new ulong[copies.Count]; + var total = 0UL; + for (var index = 0; index < copies.Count; index++) + { + var written = copies[index].Buffer.LastGpuWriteTick; + if (written == 0) + { + return false; + } + + ticks[index] = written; + placements[index] = total; + total += (copies[index].Size + BufferDownloadBatchPlanner.Alignment - 1) & ~(BufferDownloadBatchPlanner.Alignment - 1); + } + + if (total > AsyncReadbackLimit) + { + return false; + } + + buffer = new GpuBuffer(_device, _scheduler, GpuBufferUsage.Download, 0, BufferUsageFlags.TransferDstBit, total); + for (var index = 0; index < copies.Count; index++) + { + buffer.CopyFrom( + _scheduler.Current, copies[index].Buffer, copies[index].SourceOffset, placements[index], copies[index].Size, + AccessFlags.MemoryWriteBit, AccessFlags.None, AccessFlags.MemoryReadBit | AccessFlags.MemoryWriteBit, AccessFlags.HostReadBit); + } + + tick = _scheduler.Flush(); + offsets = placements; + writeTicks = ticks; + return true; + } + + private static readonly bool MainQueuePendingReads = + Environment.GetEnvironmentVariable("SHARPEMU_MAIN_QUEUE_PENDING_READS") != "0"; + + // Pending reads a guest thread re-issues before it lets the worker read synchronously. + private const int PendingReadAttempts = 4; + private bool CompleteReadMemoryOnGpu(PendingDownload pending, bool retrySynchronously) { using var readbackScope = GuestMemoryProfile.Measure(GuestMemoryProfile.Operation.BufferReadback); + if (pending.Mapped) + { + return CompleteMappedRead(pending, retrySynchronously); + } + + if (pending.Ticket is null) + { + return CompleteMainQueueRead(pending, retrySynchronously); + } + var readback = AsyncReadback ?? throw SubmissionScheduler.Fatal("The pending readback lost its queue."); try { @@ -1241,7 +2111,7 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable var copies = pending.Copies; readback.Complete(pending.Ticket, (index, bytes) => { - if (!_backing.TryWriteBacking(copies[index].Address, bytes)) + if (!WriteDownloaded(copies[index].Address, bytes)) { throw SubmissionScheduler.Fatal($"Could not write the required direct backing: addr=0x{copies[index].Address:X16} size=0x{(ulong)bytes.Length:X16}"); } @@ -1269,6 +2139,155 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable } } + // Guest buffers live in host-visible device memory on unified-memory devices (Apple GPUs), so a + // readback can copy straight out of the buffer once the last GPU write to it retired. That skips + // the copy command at the tail of the queue and the wait for every draw recorded after the writer. + // SHARPEMU_UNIFIED_BUFFERS=0 keeps device-local buffers and copy readbacks. + private static readonly bool UnifiedBuffersEnabled = + Environment.GetEnvironmentVariable("SHARPEMU_UNIFIED_BUFFERS") != "0"; + + private static bool HasUnifiedMemoryType(GpuDeviceInfo device) + { + const MemoryPropertyFlags unified = MemoryPropertyFlags.DeviceLocalBit | MemoryPropertyFlags.HostVisibleBit | MemoryPropertyFlags.HostCoherentBit; + for (uint index = 0; index < device.MemoryTypeCount; index++) + { + if ((device.GetMemoryTypeFlags(index) & unified) == unified) + { + return true; + } + } + + return false; + } + + private static bool AreMapped(List copies, out ulong writer) + { + writer = 0; + foreach (var copy in copies) + { + if (copy.Buffer.MappedPointer == null || !copy.Buffer.IsCoherent) + { + return false; + } + + writer = Math.Max(writer, copy.Buffer.LastGpuWriteTick); + } + + return true; + } + + private void ApplyMappedRead(List copies, ulong windowBegin, ulong windowEnd, ulong guestAddress, ulong size, + bool isWrite, long started, GuestMemoryProfile.ReadbackSource source) + { + foreach (var copy in copies) + { + var bytes = new ReadOnlySpan(copy.Buffer.MappedPointer + copy.SourceOffset, checked((int)copy.Size)); + if (!WriteDownloaded(copy.Address, bytes)) + { + throw SubmissionScheduler.Fatal($"Could not write the required direct backing: addr=0x{copy.Address:X16} size=0x{copy.Size:X16}"); + } + + _gpuModifiedRanges.Remove(copy.Address, copy.Size); + } + + _tracker.ClearGpuDirtyPages(windowBegin, windowEnd - windowBegin); + if (isWrite) + { + _tracker.MarkCpuDirtyPages(guestAddress, size); + } + + MappedReadbacks++; + RecordReadback(windowBegin, windowEnd, isWrite, copies, started, source); + } + + // Readbacks served from a buffer's own mapping, without a copy command. + internal long MappedReadbacks { get; private set; } + + internal bool UnifiedBuffers => _unifiedBuffers; + + private bool CompleteMappedRead(PendingDownload pending, bool retrySynchronously) + { + try + { + if (!IsPendingDownloadCurrent(pending)) + { + if (retrySynchronously) + { + PendingReadbacksRetried++; + Interlocked.Increment(ref _reportedRetried); + ReadMemoryOnGpu(pending.GuestAddress, pending.Size, pending.IsWrite, pending.Source); + } + + return false; + } + + PendingReadbacksApplied++; + Interlocked.Increment(ref _reportedApplied); + ApplyMappedRead(pending.Copies, pending.WindowBegin, pending.WindowEnd, pending.GuestAddress, pending.Size, + pending.IsWrite, pending.Started, pending.Source); + return true; + } + finally + { + foreach (var copy in pending.Copies) + { + copy.Buffer.ReleaseForeignRead(); + } + } + } + + private bool CompleteMainQueueRead(PendingDownload pending, bool retrySynchronously) + { + var buffer = pending.MainQueueBuffer ?? throw SubmissionScheduler.Fatal("The pending readback lost its buffer."); + try + { + if (!IsPendingDownloadCurrent(pending)) + { + if (retrySynchronously) + { + PendingReadbacksRetried++; + Interlocked.Increment(ref _reportedRetried); + ReadMemoryOnGpu(pending.GuestAddress, pending.Size, pending.IsWrite, pending.Source); + } + + return false; + } + + PendingReadbacksApplied++; + Interlocked.Increment(ref _reportedApplied); + var copies = pending.Copies; + buffer.Invalidate(0, buffer.Size); + for (var index = 0; index < copies.Count; index++) + { + var bytes = buffer.Mapped.Slice((int)pending.MainQueueOffsets[index], (int)copies[index].Size); + if (!WriteDownloaded(copies[index].Address, bytes)) + { + throw SubmissionScheduler.Fatal($"Could not write the required direct backing: addr=0x{copies[index].Address:X16} size=0x{copies[index].Size:X16}"); + } + + _gpuModifiedRanges.Remove(copies[index].Address, copies[index].Size); + } + + _tracker.ClearGpuDirtyPages(pending.WindowBegin, pending.WindowEnd - pending.WindowBegin); + if (pending.IsWrite) + { + _tracker.MarkCpuDirtyPages(pending.GuestAddress, pending.Size); + } + + RecordReadback(pending.WindowBegin, pending.WindowEnd, pending.IsWrite, copies, pending.Started, pending.Source); + return true; + } + finally + { + // The guest thread waited for the submission, so the GPU is done with the buffer. + buffer.Dispose(); + foreach (var copy in pending.Copies) + { + copy.Buffer.ReleaseForeignRead(); + } + } + } + private bool IsPendingDownloadCurrent(PendingDownload pending) { var total = 0UL; @@ -1314,9 +2333,12 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable private List CollectReadbackWindow(ulong guestAddress, ulong size, out ulong windowBegin, out ulong windowEnd) => CollectReadbackWindow(_registry.GetBuffer(FindBuffer(guestAddress, size)), guestAddress, size, out windowBegin, out windowEnd); - private List CollectReadbackWindow(GpuBuffer buffer, ulong guestAddress, ulong size, out ulong windowBegin, out ulong windowEnd) + private List CollectReadbackWindow(GpuBuffer buffer, ulong guestAddress, ulong size, out ulong windowBegin, out ulong windowEnd) => + CollectReadbackWindow(buffer, guestAddress, size, ReadbackWindowBytes, out windowBegin, out windowEnd); + + private List CollectReadbackWindow(GpuBuffer buffer, ulong guestAddress, ulong size, ulong windowSize, + out ulong windowBegin, out ulong windowEnd) { - const ulong windowSize = ReadbackWindowBytes; var bufferEnd = buffer.CpuAddress + buffer.Size; windowBegin = Math.Max(guestAddress & ~(windowSize - 1), buffer.CpuAddress); windowEnd = Math.Min(Math.Max(windowBegin + windowSize, guestAddress + size), bufferEnd); @@ -1364,7 +2386,7 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable _tracker.UntrackMemory(buffer.CpuAddress, buffer.Size); Unregister(bufferIdentifier); - _registry.CompleteRetirement(bufferIdentifier); + CompleteRetirementAfterSubmittedWork(bufferIdentifier); } private void WriteDataBuffer(GpuBuffer buffer, ulong address, ReadOnlySpan source) @@ -1433,6 +2455,15 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable } Unregister(bufferIdentifier); + CompleteRetirementAfterSubmittedWork(bufferIdentifier); + } + + // Unregistering clears the buffer's page-table entries only for work recorded from now + // on. Work already recorded or in flight can still reach it through its device address + // (an async readback waits only for its last recorded writer), so it is destroyed once + // that work completes. + private void CompleteRetirementAfterSubmittedWork(ResourceSlotIdentifier bufferIdentifier) + { if (_scheduler.Active) { _scheduler.QueueCompletionAction(() => _registry.CompleteRetirement(bufferIdentifier)); @@ -1444,7 +2475,7 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable } // Set when a second queue can copy readbacks; null keeps every readback on the main queue. - internal VulkanAsyncReadback? AsyncReadback { get; set; } + internal IBufferReadback? AsyncReadback { get; set; } private const ulong AsyncReadbackLimit = 64UL << 20; @@ -1590,7 +2621,7 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable readback.Read(pieces, waitTick, (index, bytes) => { - if (!_backing.TryWriteBacking(copies[index].Address, bytes)) + if (!WriteDownloaded(copies[index].Address, bytes)) { throw SubmissionScheduler.Fatal($"Could not write the required direct backing: addr=0x{copies[index].Address:X16} size=0x{(ulong)bytes.Length:X16}"); } @@ -1622,7 +2653,7 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable var placement = copy.Placement; var offset = baseOffset + placement.DataOffset; _download.Invalidate(offset, placement.DataSize); - if (!_backing.TryWriteBacking(copy.Address, _download.Mapped.Slice((int)offset, (int)placement.DataSize))) + if (!WriteDownloaded(copy.Address, _download.Mapped.Slice((int)offset, (int)placement.DataSize))) { throw SubmissionScheduler.Fatal($"Could not write the required direct backing: addr=0x{copy.Address:X16} size=0x{placement.DataSize:X16}"); } @@ -1665,6 +2696,18 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable } newBuffer.CopyFrom(_scheduler.Current, overlap, 0, overlap.CpuAddress - newBuffer.CpuAddress, overlap.Size); + _sharedPages.NoteOwnerCopy(overlap, newBuffer, _scheduler.CurrentTick); + if (Volatile.Read(ref _shaderWrittenPageCount) != 0) + { + lock (_shaderWrittenPages) + { + for (var page = overlap.CpuAddress; page < overlap.CpuAddress + overlap.Size; page += TrackerLayout.PageBytes) + { + if (_shaderWrittenPages.TryGetValue(page, out var owner) && ReferenceEquals(owner, overlap)) + _shaderWrittenPages[page] = newBuffer; + } + } + } DeleteBuffer(overlappingBufferIdentifier); } @@ -1685,9 +2728,31 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable overlapping.Add(_registry.GetRegisteredIdentifier(index)); } - var bufferIdentifier = _registry.AllocateBuffer(new GpuBuffer( - _device, _scheduler, GpuBufferUsage.DeviceLocal, overlap.Begin, - GpuBuffer.AllFlags | BufferUsageFlags.ShaderDeviceAddressBit, overlap.End - overlap.Begin, allowSlab: true), overlap.Begin, overlap.End - overlap.Begin); + GpuBuffer CreateGpuBuffer() => new( + _device, _scheduler, _unifiedBuffers ? GpuBufferUsage.Unified : GpuBufferUsage.DeviceLocal, overlap.Begin, + GpuBuffer.AllFlags | BufferUsageFlags.ShaderDeviceAddressBit, overlap.End - overlap.Begin, allowSlab: true); + GpuBuffer created; + try + { + created = CreateGpuBuffer(); + } + catch (GpuBuffer.OutOfMemoryException) + { + // Images hold most of the device memory: free the ones that can go and try once more. + RequireImageCache().ReclaimForAllocation(); + try + { + created = CreateGpuBuffer(); + } + catch (GpuBuffer.OutOfMemoryException again) + { + throw SubmissionScheduler.Fatal(again.Message); + } + } + + _device.NameObject?.Invoke(ObjectType.Buffer, created.Handle.Handle, + $"guest 0x{overlap.Begin:X}+0x{overlap.End - overlap.Begin:X}"); + var bufferIdentifier = _registry.AllocateBuffer(created, overlap.Begin, overlap.End - overlap.Begin); foreach (var oldId in overlapping) { MergeOverlappingBuffer(bufferIdentifier, oldId, !overlap.HasStreamLeap); @@ -1698,7 +2763,7 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable } private bool SynchronizeBuffer(GpuBuffer buffer, ulong guestAddress, ulong size, bool isWritten, bool isTexelBuffer, - bool preserveCpuWriteHotPages = true) + bool preserveCpuWriteHotPages = true, bool readImageBacking = false, bool skipCpuWriteHotPages = false) { using var profileScope = RenderPhaseProfile.MeasureDetail(RenderPhaseProfile.Phase.BufferDirtySynchronization); var startedAt = BufferUploadProfile.Enabled ? System.Diagnostics.Stopwatch.GetTimestamp() : 0; @@ -1713,27 +2778,46 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable profileScope.SwitchPhase(isWritten ? RenderPhaseProfile.Phase.BufferDirtySyncWritten : isTexelBuffer ? RenderPhaseProfile.Phase.BufferDirtySyncTexel : RenderPhaseProfile.Phase.BufferDirtySyncUpload); - var copies = new List(); - var totalSize = 0UL; - GpuBuffer? source = null; - _tracker.ForEachUploadRange( - guestAddress, - size, - isWritten, - (address, bytes) => + var copies = _syncCopies ??= new List(); + copies.Clear(); + var sink = new UploadSink(_uploader, buffer, copies, guestAddress, size, + readImageBacking ? _tryReadImageSource ??= TryReadImageSource : null); + _tracker.ForEachUploadRange(guestAddress, size, isWritten, ref sink, preserveCpuWriteHotPages, skipCpuWriteHotPages); + var source = sink.Source; + if (source != null && _sharedPages.Count != 0) + { + // Shared pages copy only the bytes the CPU changed, so shader bytes in the buffer stay. + _sharedPages.FilterUpload(copies, buffer, buffer.CpuAddress, source.Mapped, _scheduler.CurrentTick); + if (copies.Count == 0) { - copies.Add(new BufferCopy(totalSize, buffer.Offset(address), bytes)); - totalSize += bytes; - }, - () => source = _uploader.PrepareSource(buffer.CpuAddress, CollectionsMarshal.AsSpan(copies), totalSize, guestAddress, size), - preserveCpuWriteHotPages); + source = null; + } + } + if (source != null) { + if (WatchedRanges.Length != 0) + { + foreach (var copy in copies) + WatchEvent(buffer.CpuAddress + copy.DstOffset, copy.Size, "upload"); + } + buffer.NoteGpuWrite(); var command = _scheduler.Current; command.EndRendering(); var native = new CommandBuffer(command.Handle); var vk = _device.Vk; + if (_uploadBatchActive) + { + OpenUploadBatch(native); + var batched = CollectionsMarshal.AsSpan(copies); + fixed (BufferCopy* pointer = batched) + { + vk.CmdCopyBuffer(native, source.Handle, buffer.Handle, (uint)batched.Length, pointer); + } + } + else + { var before = new BufferMemoryBarrier2 { SType = StructureType.BufferMemoryBarrier2, @@ -1760,6 +2844,7 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable VulkanSynchronization.PipelineBarrier(vk, native, PipelineStageFlags.TransferBit, PipelineStageFlags.AllCommandsBit, DependencyFlags.ByRegionBit, 0, null, 1, &after, 0, null); + } } if (BufferUploadProfile.Enabled) @@ -1768,10 +2853,10 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable ulong hotBytes = 0; foreach (var copy in copies) hotBytes += _tracker.CountCpuWriteHotBytes(buffer.CpuAddress + copy.DstOffset, copy.Size); - BufferUploadProfile.Record(guestAddress, size, copies.Count, totalSize, hotBytes, elapsedTicks); + BufferUploadProfile.Record(guestAddress, size, copies.Count, sink.TotalSize, hotBytes, elapsedTicks); } - if (isTexelBuffer) + if (isTexelBuffer && !readImageBacking) { var copiedFromImage = RequireImageCache().TrySynchronizeBufferFromImage(buffer, guestAddress, size); if (copiedFromImage) @@ -1785,6 +2870,82 @@ public sealed unsafe class GuestBufferCache : IGuestBufferStore, IDisposable return false; } + // A sweep uploads many buffers back to back: one global barrier before the first copy and one + // after the last replace a barrier pair per buffer, so the copies share one transfer pass. + private bool _uploadBatchActive; + private ulong _uploadBatchCommand; + + private void BeginUploadBatch() + { + _uploadBatchActive = true; + _uploadBatchCommand = 0; + } + + private void OpenUploadBatch(CommandBuffer native) + { + if (_uploadBatchCommand == (ulong)native.Handle) + { + return; + } + + // A new command buffer (or the first copy) orders the copies after all earlier work. + var barrier = new MemoryBarrier2 + { + SType = StructureType.MemoryBarrier2, + SrcAccessMask = AccessFlags2.MemoryReadBit | AccessFlags2.MemoryWriteBit, + DstAccessMask = AccessFlags2.TransferWriteBit, + }; + VulkanSynchronization.PipelineBarrier(_device.Vk, + native, PipelineStageFlags.AllCommandsBit, PipelineStageFlags.TransferBit, 0, 1, &barrier, 0, null, 0, null); + _uploadBatchCommand = (ulong)native.Handle; + } + + private void EndUploadBatch() + { + _uploadBatchActive = false; + if (_uploadBatchCommand == 0) + { + return; + } + + // Recorded on the current command buffer; submission order covers copies recorded on an + // earlier one. + var command = _scheduler.Current; + command.EndRendering(); + var barrier = new MemoryBarrier2 + { + SType = StructureType.MemoryBarrier2, + SrcAccessMask = AccessFlags2.TransferWriteBit, + DstAccessMask = AccessFlags2.MemoryReadBit | AccessFlags2.MemoryWriteBit, + }; + VulkanSynchronization.PipelineBarrier(_device.Vk, + new CommandBuffer(command.Handle), PipelineStageFlags.TransferBit, PipelineStageFlags.AllCommandsBit, 0, 1, &barrier, 0, null, 0, null); + _uploadBatchCommand = 0; + } + + // The copies of one synchronization; the tracker forbids nesting, so one list per thread is reused. + [ThreadStatic] + private static List? _syncCopies; + + // Collects a buffer's CPU-dirty runs as copies, then stages them while the tracker holds the regions. + private struct UploadSink(GuestBufferUploader uploader, GpuBuffer buffer, List copies, ulong guestAddress, ulong size, + GuestBufferSourceReader? readSource) + : GuestPageTracker.IUploadRangeSink + { + public ulong TotalSize { get; private set; } + + public GpuBuffer? Source { get; private set; } + + public void Range(ulong address, ulong bytes) + { + copies.Add(new BufferCopy(TotalSize, buffer.Offset(address), bytes)); + TotalSize += bytes; + } + + public void Upload() => + Source = uploader.PrepareSource(buffer.CpuAddress, CollectionsMarshal.AsSpan(copies), TotalSize, guestAddress, size, readSource); + } + private GpuBuffer? FindOwner(ulong guestAddress, ulong size) { return _registry.TryGetRegisteredBuffer(_registry.FindContainingBuffer(guestAddress, size)); diff --git a/src/SharpEmu.Libs/Gpu/Buffers/GuestBufferRegistry.cs b/src/SharpEmu.Libs/Gpu/Buffers/GuestBufferRegistry.cs index 2a32902..e98ffc1 100644 --- a/src/SharpEmu.Libs/Gpu/Buffers/GuestBufferRegistry.cs +++ b/src/SharpEmu.Libs/Gpu/Buffers/GuestBufferRegistry.cs @@ -47,6 +47,11 @@ internal sealed class GuestBufferRegistry : IDisposable where TBuffer : public int RegisteredCount => _registered.Count; public ulong RegisteredBytes { get; private set; } + private ulong _residentBytes; + private ulong _retiredBytes; + // Lookup removal does not release the GPU resource. Count it until disposal. + public ulong ResidentBytes => Interlocked.Read(ref _residentBytes); + public ulong RetiredBytes => Interlocked.Read(ref _retiredBytes); public ResourceSlotIdentifier AllocateBuffer(TBuffer resource, ulong address, ulong size) { @@ -69,6 +74,7 @@ internal sealed class GuestBufferRegistry : IDisposable where TBuffer : entry.State = BufferLifetimeState.Allocated; entry.LastUseTick = 0; } + Interlocked.Add(ref _residentBytes, size); return identifier; } @@ -137,6 +143,7 @@ internal sealed class GuestBufferRegistry : IDisposable where TBuffer : _registered.Remove(entry.Address); _recency.Remove(entry.RecencyNode!); RegisteredBytes -= entry.Size; + Interlocked.Add(ref _retiredBytes, entry.Size); entry.State = BufferLifetimeState.Retiring; } @@ -219,11 +226,14 @@ internal sealed class GuestBufferRegistry : IDisposable where TBuffer : private void ReleaseResource(Entry entry) { var resource = entry.Resource; + var retiring = entry.State == BufferLifetimeState.Retiring; entry.Resource = null; entry.State = BufferLifetimeState.Released; if (!_disposed && entry.Identifier.Generation != uint.MaxValue) _availableIndices.Push((int)entry.Identifier.Index); resource?.Dispose(); + Interlocked.Add(ref _residentBytes, unchecked(0UL - entry.Size)); + if (retiring) Interlocked.Add(ref _retiredBytes, unchecked(0UL - entry.Size)); } private Entry? FindEntry(ResourceSlotIdentifier identifier) => diff --git a/src/SharpEmu.Libs/Gpu/Buffers/GuestBufferUploader.cs b/src/SharpEmu.Libs/Gpu/Buffers/GuestBufferUploader.cs index 8dc363a..dd3356e 100644 --- a/src/SharpEmu.Libs/Gpu/Buffers/GuestBufferUploader.cs +++ b/src/SharpEmu.Libs/Gpu/Buffers/GuestBufferUploader.cs @@ -8,12 +8,14 @@ using Silk.NET.Vulkan; namespace SharpEmu.Libs.Gpu.Buffers; +internal delegate bool GuestBufferSourceReader(ulong address, Span destination); + // The cache owns the staging ring; temporary sources retire after their submission. internal sealed class GuestBufferUploader( GpuDeviceInfo device, SubmissionScheduler scheduler, ICpuMemory guest, GpuRingBuffer staging) { public GpuBuffer? PrepareSource(ulong bufferAddress, Span regions, - ulong totalSize, ulong requestedAddress, ulong requestedSize) + ulong totalSize, ulong requestedAddress, ulong requestedSize, GuestBufferSourceReader? readSource = null) { using var profileScope = RenderPhaseProfile.MeasureDetail(RenderPhaseProfile.Phase.BufferStagingUpload); if (regions.IsEmpty) @@ -26,7 +28,7 @@ internal sealed class GuestBufferUploader( foreach (ref var copy in regions) { ReadGuestBytes(bufferAddress + copy.DstOffset, - staging.Mapped.Slice((int)(baseOffset + copy.SrcOffset), (int)copy.Size), requestedAddress, requestedSize); + staging.Mapped.Slice((int)(baseOffset + copy.SrcOffset), (int)copy.Size), requestedAddress, requestedSize, readSource); copy.SrcOffset += baseOffset; } @@ -38,7 +40,7 @@ internal sealed class GuestBufferUploader( foreach (ref readonly var copy in regions) { ReadGuestBytes(bufferAddress + copy.DstOffset, - temporary.Mapped.Slice((int)copy.SrcOffset, (int)copy.Size), requestedAddress, requestedSize); + temporary.Mapped.Slice((int)copy.SrcOffset, (int)copy.Size), requestedAddress, requestedSize, readSource); } temporary.Flush(0, totalSize); @@ -46,9 +48,9 @@ internal sealed class GuestBufferUploader( return temporary; } - private void ReadGuestBytes(ulong address, Span destination, ulong requestedAddress, ulong requestedSize) + private void ReadGuestBytes(ulong address, Span destination, ulong requestedAddress, ulong requestedSize, GuestBufferSourceReader? readSource) { - if (!guest.TryRead(address, destination)) + if (!(readSource is null ? guest.TryRead(address, destination) : readSource(address, destination))) { Console.Error.WriteLine($"[GPU][READ_FAILURE] addr=0x{address:X16} size=0x{destination.Length:X}"); Console.Error.WriteLine($"[GPU][READ_FAILURE] binding=0x{requestedAddress:X16} size=0x{requestedSize:X}"); diff --git a/src/SharpEmu.Libs/Gpu/Buffers/IDeviceMemoryAllocator.cs b/src/SharpEmu.Libs/Gpu/Buffers/IDeviceMemoryAllocator.cs new file mode 100644 index 0000000..6988784 --- /dev/null +++ b/src/SharpEmu.Libs/Gpu/Buffers/IDeviceMemoryAllocator.cs @@ -0,0 +1,12 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using Silk.NET.Vulkan; + +namespace SharpEmu.Libs.Gpu.Buffers; + +public interface IDeviceMemoryAllocator +{ + Result AllocateMemory(in MemoryAllocateInfo info, out DeviceMemory memory); + void FreeMemory(DeviceMemory memory); +} diff --git a/src/SharpEmu.Libs/Gpu/Buffers/IGuestImageCache.cs b/src/SharpEmu.Libs/Gpu/Buffers/IGuestImageCache.cs index e6b247b..73f6972 100644 --- a/src/SharpEmu.Libs/Gpu/Buffers/IGuestImageCache.cs +++ b/src/SharpEmu.Libs/Gpu/Buffers/IGuestImageCache.cs @@ -11,6 +11,8 @@ public interface IGuestImageCache { ImageRegionInfo QueryRegion(ulong address, ulong size); + bool HasGpuModifiedImageBytes(ulong address, ulong size) => QueryRegion(address, size).GpuImageBytes; + bool ClearMetadata(ulong address); bool OverlapsDccMetadata(ulong address, ulong size); @@ -19,5 +21,11 @@ public interface IGuestImageCache void InvalidateMemoryFromGpu(ulong address, ulong size); + // A possible shader write through a raw buffer: only images that copy memory become stale. + void InvalidateMemoryCopiesFromGpu(ulong address, ulong size) { } + bool TrySynchronizeBufferFromImage(GpuBuffer buffer, ulong address, ulong size); + + // Frees every image that can go and waits for retired ones, after an allocation failed. + void ReclaimForAllocation(); } diff --git a/src/SharpEmu.Libs/Gpu/Buffers/OptimalImageMemoryPool.cs b/src/SharpEmu.Libs/Gpu/Buffers/OptimalImageMemoryPool.cs new file mode 100644 index 0000000..c8d0165 --- /dev/null +++ b/src/SharpEmu.Libs/Gpu/Buffers/OptimalImageMemoryPool.cs @@ -0,0 +1,297 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using Silk.NET.Vulkan; + +namespace SharpEmu.Libs.Gpu.Buffers; + +// A place in device memory an image is bound to: its own allocation, or a range of a block. +public readonly record struct OptimalImageMemoryAllocation(DeviceMemory Memory, ulong Offset, ulong Size, int Block, uint MemoryType = 0); + +// Images share large device-memory blocks instead of one allocation each: a driver caps the +// number of live allocations (4096 on common Windows drivers) far below the number of textures +// a bindless title keeps resident. Images are all optimal-tiled, so no linear/optimal +// granularity separates them; each range only honours its own alignment. +public sealed unsafe class OptimalImageMemoryPool +{ + public const ulong BlockSize = 256UL << 20; + public const ulong DedicatedThreshold = 64UL << 20; + + // Freed memory kept for the next image instead of going back to the driver. The image + // cache recreates images all the time (a target that grows, overlapping views) mostly at + // the same sizes; handing each one back and allocating it again costs a driver call and + // a zeroed allocation every time, and capture tools keep every freed allocation alive. + public const int MaxEmptyBlocks = 4; + public const ulong RetainedDedicatedLimit = 1UL << 30; + + private sealed class Block + { + public required DeviceMemory Memory; + public required uint MemoryType; + public required ulong Size; + public readonly SortedDictionary Free = new(); + public ulong Used; + } + + private readonly IDeviceMemoryAllocator _device; + private readonly ulong _blockSize; + private readonly bool _deviceAddress; + private readonly List _blocks = []; + private readonly List<(DeviceMemory Memory, ulong Size, uint MemoryType)> _retainedDedicated = []; + private ulong _retainedDedicatedBytes; + private readonly object _gate = new(); + private long _allocatedBytes; + private long _placedBytes; + + // `deviceAddress` allocates the blocks with VK_MEMORY_ALLOCATE_DEVICE_ADDRESS_BIT, so + // buffers with a shader device address can live in them. + public OptimalImageMemoryPool(IDeviceMemoryAllocator device, ulong blockSize = BlockSize, bool deviceAddress = false) + { + _device = device; + _blockSize = blockSize; + _deviceAddress = deviceAddress; + } + + // Bytes reserved from the Vulkan driver, including free space left inside + // pooled blocks and dedicated allocations still held by the pool. + public ulong AllocatedBytes => (ulong)Math.Max(Volatile.Read(ref _allocatedBytes), 0); + + // Bytes occupied by live image placements. This excludes free ranges inside + // pooled blocks and is therefore the useful fragmentation comparison. + public ulong PlacedBytes => (ulong)Math.Max(Volatile.Read(ref _placedBytes), 0); + + public ulong UnusedBytes + { + get + { + lock (_gate) return checked((ulong)(_allocatedBytes - _placedBytes)); + } + } + + // Ranges placed in blocks, and blocks held from the driver (empty ones included). + public int Placements => Volatile.Read(ref _placements); + + public int Blocks + { + get + { + lock (_gate) + { + return _blocks.Count(block => block is not null); + } + } + } + + private int _placements; + + public Result Allocate(in MemoryRequirements requirements, uint memoryType, out OptimalImageMemoryAllocation memory) + { + memory = default; + if (requirements.Size >= DedicatedThreshold) + { + lock (_gate) + { + var size = requirements.Size; + var retained = _retainedDedicated.FindIndex(entry => entry.Size == size && entry.MemoryType == memoryType); + if (retained >= 0) + { + var entry = _retainedDedicated[retained]; + _retainedDedicated.RemoveAt(retained); + _retainedDedicatedBytes -= entry.Size; + memory = new OptimalImageMemoryAllocation(entry.Memory, 0, entry.Size, -1, entry.MemoryType); + Interlocked.Add(ref _placedBytes, checked((long)entry.Size)); + return Result.Success; + } + } + + var info = new MemoryAllocateInfo { SType = StructureType.MemoryAllocateInfo, AllocationSize = requirements.Size, MemoryTypeIndex = memoryType }; + var dedicated = _device.AllocateMemory(info, out var handle); + if (dedicated == Result.Success) + { + memory = new OptimalImageMemoryAllocation(handle, 0, requirements.Size, -1, memoryType); + Interlocked.Add(ref _allocatedBytes, checked((long)requirements.Size)); + Interlocked.Add(ref _placedBytes, checked((long)requirements.Size)); + } + + return dedicated; + } + + var alignment = Math.Max(requirements.Alignment, 1UL); + lock (_gate) + { + for (var index = 0; index < _blocks.Count; index++) + { + if (_blocks[index] is { } block && block.MemoryType == memoryType && TryPlace(block, requirements.Size, alignment, out var offset)) + { + memory = new OptimalImageMemoryAllocation(block.Memory, offset, requirements.Size, index); + Interlocked.Increment(ref _placements); + Interlocked.Add(ref _placedBytes, checked((long)requirements.Size)); + return Result.Success; + } + } + + var addressFlags = new MemoryAllocateFlagsInfo { SType = StructureType.MemoryAllocateFlagsInfo, Flags = MemoryAllocateFlags.DeviceAddressBit }; + var blockInfo = new MemoryAllocateInfo + { + SType = StructureType.MemoryAllocateInfo, + PNext = _deviceAddress ? &addressFlags : null, + AllocationSize = Math.Max(_blockSize, requirements.Size), + MemoryTypeIndex = memoryType, + }; + var result = _device.AllocateMemory(blockInfo, out var blockMemory); + if (result == Result.ErrorOutOfDeviceMemory && blockInfo.AllocationSize > requirements.Size) + { + // A pooling policy must not reject an image whose actual storage + // still fits. Keep all allocation flags, but request its exact size. + blockInfo.AllocationSize = requirements.Size; + result = _device.AllocateMemory(blockInfo, out blockMemory); + } + if (result != Result.Success) + { + return result; + } + + var created = new Block { Memory = blockMemory, MemoryType = memoryType, Size = blockInfo.AllocationSize }; + created.Free.Add(0, blockInfo.AllocationSize); + var slot = _blocks.IndexOf(null); + if (slot < 0) + { + slot = _blocks.Count; + _blocks.Add(created); + } + else + { + _blocks[slot] = created; + } + + TryPlace(created, requirements.Size, alignment, out var placed); + Interlocked.Add(ref _allocatedBytes, checked((long)blockInfo.AllocationSize)); + Interlocked.Add(ref _placedBytes, checked((long)requirements.Size)); + memory = new OptimalImageMemoryAllocation(blockMemory, placed, requirements.Size, slot); + Interlocked.Increment(ref _placements); + return Result.Success; + } + } + + public void Free(in OptimalImageMemoryAllocation memory) + { + if (memory.Memory.Handle == 0) + { + return; + } + + if (memory.Block < 0) + { + Interlocked.Add(ref _placedBytes, -checked((long)memory.Size)); + lock (_gate) + { + _retainedDedicated.Add((memory.Memory, memory.Size, memory.MemoryType)); + _retainedDedicatedBytes += memory.Size; + // Past the limit the oldest retained allocations go back to the driver. + while (_retainedDedicatedBytes > RetainedDedicatedLimit) + { + var oldest = _retainedDedicated[0]; + _retainedDedicated.RemoveAt(0); + _retainedDedicatedBytes -= oldest.Size; + _device.FreeMemory(oldest.Memory); + Interlocked.Add(ref _allocatedBytes, -checked((long)oldest.Size)); + } + } + + return; + } + + lock (_gate) + { + var block = _blocks[memory.Block] ?? throw new InvalidOperationException("The image memory block was already released."); + var start = memory.Offset; + var end = memory.Offset + memory.Size; + + // Merge with the free ranges on either side. + foreach (var (freeStart, freeLength) in block.Free) + { + if (freeStart + freeLength == start) + { + start = freeStart; + block.Free.Remove(freeStart); + break; + } + } + + if (block.Free.TryGetValue(end, out var nextLength)) + { + block.Free.Remove(end); + end += nextLength; + } + + block.Free[start] = end - start; + block.Used -= memory.Size; + Interlocked.Decrement(ref _placements); + Interlocked.Add(ref _placedBytes, -checked((long)memory.Size)); + // An emptied block stays for the next images while few enough blocks are empty. + if (block.Used == 0 && _blocks.Count(candidate => candidate is { Used: 0 }) > MaxEmptyBlocks) + { + _device.FreeMemory(block.Memory); + _blocks[memory.Block] = null; + Interlocked.Add(ref _allocatedBytes, -checked((long)block.Size)); + } + } + } + + // Hands every retained allocation and empty block back to the driver, for memory pressure. + public void ReleaseRetained() + { + lock (_gate) + { + foreach (var (retained, size, _) in _retainedDedicated) + { + _device.FreeMemory(retained); + Interlocked.Add(ref _allocatedBytes, -checked((long)size)); + } + + _retainedDedicated.Clear(); + _retainedDedicatedBytes = 0; + for (var index = 0; index < _blocks.Count; index++) + { + if (_blocks[index] is { Used: 0 } empty) + { + _device.FreeMemory(empty.Memory); + _blocks[index] = null; + Interlocked.Add(ref _allocatedBytes, -checked((long)empty.Size)); + } + } + } + } + + private static bool TryPlace(Block block, ulong size, ulong alignment, out ulong offset) + { + foreach (var (start, length) in block.Free) + { + var aligned = (start + alignment - 1) / alignment * alignment; + var padding = aligned - start; + if (padding > length || length - padding < size) + { + continue; + } + + block.Free.Remove(start); + if (padding != 0) + { + block.Free[start] = padding; + } + + var tail = length - padding - size; + if (tail != 0) + { + block.Free[aligned + size] = tail; + } + + block.Used += size; + offset = aligned; + return true; + } + + offset = 0; + return false; + } +} diff --git a/src/SharpEmu.Libs/Gpu/Buffers/SharedPageShadows.cs b/src/SharpEmu.Libs/Gpu/Buffers/SharedPageShadows.cs new file mode 100644 index 0000000..da10cc8 --- /dev/null +++ b/src/SharpEmu.Libs/Gpu/Buffers/SharedPageShadows.cs @@ -0,0 +1,301 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using Silk.NET.Vulkan; + +namespace SharpEmu.Libs.Gpu.Buffers; + +// Pages the CPU and shaders both write. On the console both land in one memory. Here the page +// tracker gives each page a single owner, so a shader store through a device address into a page +// the CPU dirtied cannot make it GPU-owned: the next upload copied the whole guest page, stale +// where the shader wrote, over the GPU copy, and the guest never saw the shader's bytes. +// +// A page shaders wrote through device addresses gets a shadow once the CPU touches it (up to the +// capacity): the bytes both copies last agreed on. An upload copies only the bytes the CPU +// changed since (the shader's bytes in the GPU copy stay), and the bytes the GPU +// changed since are merged into guest memory where the CPU left them alone. A byte both changed +// keeps the CPU's value in guest memory, and the next upload carries it to the GPU copy. +internal sealed class SharedPageShadows(ulong pageBytes, int capacity) +{ + private sealed class Shadow(object owner, byte[] bytes) + { + public object Owner = owner; + public readonly byte[] Bytes = bytes; + // The submission tick of the last upload staged for the page; the GPU copy lacks + // its bytes until that tick completes. + public ulong StagedTick; + } + + private readonly Dictionary _pages = new(); + private readonly object _gate = new(); + private int _count; + private long _mergedBytes; + private long _skippedUploadBytes; + + public int Count => Volatile.Read(ref _count); + + // Bytes the GPU wrote that reached guest memory through a merge. + public long MergedBytes => Interlocked.Read(ref _mergedBytes); + + // Bytes uploads left alone because the CPU had not changed them. + public long SkippedUploadBytes => Interlocked.Read(ref _skippedUploadBytes); + + public bool Contains(ulong page) + { + if (Count == 0) + { + return false; + } + + lock (_gate) + { + return _pages.ContainsKey(page); + } + } + + // Starts shadowing a page from bytes both copies hold, which uploads staged up to tick may + // still change. A shader-written page is adopted when the CPU first touches it: that access + // downloads the page, so guest memory then matches the GPU copy. A page the CPU dirtied before + // its first shader write was reported is adopted from the GPU copy instead; it differs where + // the CPU wrote and where the shader wrote, the next upload copies both, and those shader + // bytes are lost once. False when the capacity is reached. + public bool Adopt(ulong page, object owner, ReadOnlySpan gpuPage, ulong tick = 0) + { + RequirePage(gpuPage.Length); + lock (_gate) + { + if (_pages.ContainsKey(page)) + { + return true; + } + + if (_pages.Count >= capacity) + { + return false; + } + + _pages.Add(page, new Shadow(owner, gpuPage.ToArray()) { StagedTick = tick }); + Volatile.Write(ref _count, _pages.Count); + return true; + } + } + + // The cache copied the entire old buffer into its replacement. Its shared pages retain + // the same baseline; treating the replacement as unrelated would upload stale GPU bytes. + public void NoteOwnerCopy(object source, object target, ulong tick) + { + if (Count == 0) return; + lock (_gate) + { + foreach (var shadow in _pages.Values) + { + if (!ReferenceEquals(shadow.Owner, source)) continue; + shadow.Owner = target; + shadow.StagedTick = Math.Max(shadow.StagedTick, tick); + } + } + } + + // A download wrote GPU bytes into guest memory; on shadowed pages both copies now agree there. + public void NoteDownloaded(ulong address, ReadOnlySpan bytes) + { + if (Count == 0 || bytes.IsEmpty) + { + return; + } + + var end = address + (ulong)bytes.Length; + lock (_gate) + { + for (var page = address & ~(pageBytes - 1); page < end; page += pageBytes) + { + if (!_pages.TryGetValue(page, out var shadow)) + { + continue; + } + + var from = Math.Max(page, address); + var to = Math.Min(page + pageBytes, end); + bytes.Slice((int)(from - address), (int)(to - from)).CopyTo(shadow.Bytes.AsSpan((int)(from - page))); + } + } + } + + // Merges the bytes the GPU changed since the shadow into guestPage, where the CPU did not + // change them, and returns the number of merged bytes (writeGuest receives each merged run as + // an offset and length). Only once every upload staged for the page has executed: before that + // the GPU copy still lacks staged CPU bytes, which would read as GPU changes. A page whose + // owner changed is shadowed again from the new copy. gpuPage must not change during the call. + public int PullGpuWrites(ulong page, object owner, ReadOnlySpan gpuPage, Span guestPage, ulong completedTick, + Action writeGuest) + { + RequirePage(gpuPage.Length); + RequirePage(guestPage.Length); + lock (_gate) + { + if (!_pages.TryGetValue(page, out var shadow)) + { + return 0; + } + + if (!ReferenceEquals(shadow.Owner, owner)) + { + shadow.Owner = owner; + gpuPage.CopyTo(shadow.Bytes); + return 0; + } + + if (shadow.StagedTick > completedTick) + { + return 0; + } + + var bytes = shadow.Bytes.AsSpan(); + var merged = 0; + var index = 0; + while ((index += gpuPage[index..].CommonPrefixLength(bytes[index..])) < bytes.Length) + { + // A run of GPU changes; within it, the bytes the CPU left alone go to guest memory. + var runEnd = index; + while (runEnd < bytes.Length && gpuPage[runEnd] != bytes[runEnd]) + { + runEnd++; + } + + var write = -1; + for (var at = index; at <= runEnd; at++) + { + var take = at < runEnd && guestPage[at] == bytes[at]; + if (take) + { + guestPage[at] = gpuPage[at]; + write = write < 0 ? at : write; + } + else if (write >= 0) + { + writeGuest(write, at - write); + merged += at - write; + write = -1; + } + } + + gpuPage[index..runEnd].CopyTo(bytes[index..]); + index = runEnd; + } + + if (merged != 0) + { + Interlocked.Add(ref _mergedBytes, merged); + } + + return merged; + } + } + + // Narrows an upload to the bytes the CPU changed on shadowed pages. copies stage from + // staged (indexed by SrcOffset) into a buffer at targetAddress (indexed by DstOffset); copies + // over shadowed pages are split into the runs that differ from the shadow, which takes them. + // An upload into a new owner copies everything and shadows the page from it. + public void FilterUpload(List copies, object target, ulong targetAddress, ReadOnlySpan staged, ulong tick) + { + if (Count == 0) + { + return; + } + + lock (_gate) + { + List? filtered = null; + for (var index = 0; index < copies.Count; index++) + { + var copy = copies[index]; + var begin = targetAddress + copy.DstOffset; + var end = begin + copy.Size; + var emitted = begin; + for (var page = begin & ~(pageBytes - 1); page < end; page += pageBytes) + { + if (!_pages.TryGetValue(page, out var shadow)) + { + continue; + } + + if (filtered is null) + { + filtered = new List(copies.Count + 8); + for (var earlier = 0; earlier < index; earlier++) + { + filtered.Add(copies[earlier]); + } + } + + var from = Math.Max(page, begin); + var to = Math.Min(page + pageBytes, end); + if (emitted < from) + { + filtered.Add(Slice(copy, begin, emitted, from - emitted)); + } + + emitted = to; + var source = staged.Slice(checked((int)(copy.SrcOffset + (from - begin))), checked((int)(to - from))); + var shadowBytes = shadow.Bytes.AsSpan(checked((int)(from - page)), source.Length); + if (!ReferenceEquals(shadow.Owner, target)) + { + shadow.Owner = target; + filtered.Add(Slice(copy, begin, from, to - from)); + shadow.StagedTick = Math.Max(shadow.StagedTick, tick); + source.CopyTo(shadowBytes); + continue; + } + + var skipped = source.Length; + var at = 0; + while ((at += source[at..].CommonPrefixLength(shadowBytes[at..])) < source.Length) + { + var runEnd = at; + while (runEnd < source.Length && source[runEnd] != shadowBytes[runEnd]) + { + runEnd++; + } + + filtered.Add(Slice(copy, begin, from + (ulong)at, (ulong)(runEnd - at))); + // Only a copy that survives filtering can leave staged bytes missing + // from the GPU mapping. An eliminated upload has no completion to wait for. + shadow.StagedTick = Math.Max(shadow.StagedTick, tick); + source[at..runEnd].CopyTo(shadowBytes[at..]); + skipped -= runEnd - at; + at = runEnd; + } + + Interlocked.Add(ref _skippedUploadBytes, skipped); + } + + if (filtered is null) + { + continue; + } + + if (emitted < end) + { + filtered.Add(Slice(copy, begin, emitted, end - emitted)); + } + } + + if (filtered is not null) + { + copies.Clear(); + copies.AddRange(filtered); + } + } + } + + private static BufferCopy Slice(BufferCopy copy, ulong begin, ulong address, ulong size) => + new(copy.SrcOffset + (address - begin), copy.DstOffset + (address - begin), size); + + private void RequirePage(int length) + { + if ((ulong)length != pageBytes) + { + throw new ArgumentException("A shadowed page span must cover exactly one page."); + } + } +} diff --git a/src/SharpEmu.Libs/Gpu/GpuCommands/CommandStreamQueue.cs b/src/SharpEmu.Libs/Gpu/GpuCommands/CommandStreamQueue.cs index 1db6ec3..c425f39 100644 --- a/src/SharpEmu.Libs/Gpu/GpuCommands/CommandStreamQueue.cs +++ b/src/SharpEmu.Libs/Gpu/GpuCommands/CommandStreamQueue.cs @@ -582,10 +582,17 @@ public sealed class CommandStreamQueue } // Processes to empty; with the flag, heads still blocked after a full retry cycle are cancelled. - public IdleOutcome DrainForShutdown(bool cancelBlockedOnNoProgress) + // Once abandon reports true (the guest memory is gone), the remaining work is dropped unexecuted. + public IdleOutcome DrainForShutdown(bool cancelBlockedOnNoProgress, Func? abandon = null) { for (;;) { + if (abandon?.Invoke() == true) + { + DiscardPending(); + return Outcome; + } + var result = ProcessOne(); switch (result) { @@ -635,6 +642,34 @@ public sealed class CommandStreamQueue return progressed; } + private void DiscardPending() + { + lock (_gate) + { + foreach (var queue in _queues) + { + if (queue.Count == 0) + { + continue; + } + + _submissionCount -= queue.Count; + foreach (var dropped in queue) + { + if (dropped.Kind == CommandSubmissionKind.FrameBoundary) + { + _pendingBoundaries--; + } + } + + queue.Clear(); + _outcome = IdleOutcome.Cancelled; + } + + Monitor.PulseAll(_gate); + } + } + private void CancelBlocked() { lock (_gate) diff --git a/src/SharpEmu.Libs/Gpu/GpuCommands/GpuCommandInterpreter.Draws.cs b/src/SharpEmu.Libs/Gpu/GpuCommands/GpuCommandInterpreter.Draws.cs index af8fbb6..becffda 100644 --- a/src/SharpEmu.Libs/Gpu/GpuCommands/GpuCommandInterpreter.Draws.cs +++ b/src/SharpEmu.Libs/Gpu/GpuCommands/GpuCommandInterpreter.Draws.cs @@ -103,6 +103,16 @@ public sealed partial class GpuCommandInterpreter private void DrawIndirectArguments(ulong packetAddress, uint opcode, ulong argumentsAddress, bool indexed) { + if (!indexed && _host.ResolvesNonIndexedIndirectDrawOnGpu) + { + // The host reads the four arguments on the GPU, so a GPU-written argument buffer + // costs no wait for the GPU here; it reads them itself when the draw needs counts. + _deferredInstanceCountAddress = argumentsAddress + 4; + _host.DrawAuto(SubmitId, new DrawAutoArguments(packetAddress, opcode, 1, 1, 0, 0, + DrawOffsetSource.IndirectArguments, argumentsAddress)); + return; + } + if (!indexed) { var vertexCount = ReadDword(argumentsAddress); diff --git a/src/SharpEmu.Libs/Gpu/GpuCommands/GpuCommandInterpreter.Events.cs b/src/SharpEmu.Libs/Gpu/GpuCommands/GpuCommandInterpreter.Events.cs index 47af475..e6be64c 100644 --- a/src/SharpEmu.Libs/Gpu/GpuCommands/GpuCommandInterpreter.Events.cs +++ b/src/SharpEmu.Libs/Gpu/GpuCommands/GpuCommandInterpreter.Events.cs @@ -16,11 +16,11 @@ public sealed partial class GpuCommandInterpreter var eventIndex = (payload[0] >> 8) & 0x7u; var eventType = payload[0] & 0x3Fu; ulong eventAddress = 0; - if (eventType == 0x39) + if (eventType == 0x39 || (eventType == 0x38 && packet.Length == 4)) { if (packet.Length != 4) { - throw _host.Fatal($"The occlusion event packet length is not supported: length={packet.Length} address=0x{packet.PacketAddress:X16}."); + throw _host.Fatal($"The addressed event packet length is not supported: length={packet.Length} address=0x{packet.PacketAddress:X16}."); } eventAddress = Address(payload[1], payload[2]); @@ -211,9 +211,25 @@ public sealed partial class GpuCommandInterpreter } } - internal uint PushMarkerPacket(in PacketContext packet, ReadOnlySpan payload) => packet.Length - 1; + internal uint PushMarkerPacket(in PacketContext packet, ReadOnlySpan payload) + { + if (Images.ImageDropTrace.Enabled) + { + Images.ImageDropTrace.PushMarker(this, payload); + } - internal uint PopMarkerPacket(in PacketContext packet, ReadOnlySpan payload) => packet.Length - 1; + return packet.Length - 1; + } + + internal uint PopMarkerPacket(in PacketContext packet, ReadOnlySpan payload) + { + if (Images.ImageDropTrace.Enabled) + { + Images.ImageDropTrace.PopMarker(this); + } + + return packet.Length - 1; + } internal uint FlipPacket(in PacketContext packet, ReadOnlySpan payload) { diff --git a/src/SharpEmu.Libs/Gpu/GpuCommands/GpuCommandInterpreter.Labels.cs b/src/SharpEmu.Libs/Gpu/GpuCommands/GpuCommandInterpreter.Labels.cs index 4cb3dac..c15fb34 100644 --- a/src/SharpEmu.Libs/Gpu/GpuCommands/GpuCommandInterpreter.Labels.cs +++ b/src/SharpEmu.Libs/Gpu/GpuCommands/GpuCommandInterpreter.Labels.cs @@ -435,15 +435,21 @@ public sealed partial class GpuCommandInterpreter throw _host.Fatal($"The occlusion-counter dump is invalid: index=0x{eventIndex:X8} address=0x{eventAddress:X16}."); } - // Publish visible occlusion results for each depth block. + // Until real sample queries are implemented, conservatively report visibility. + // A one-sample delta can disappear when the guest scales or reduces the counts. + const ulong visibleCounterStep = 0x2FFFFFF; const ulong readyBit = 1UL << 63; var result = readyBit | SyntheticOcclusionCounter; for (var depthBlock = 0u; depthBlock < 16u; depthBlock++) { - WriteQword(eventAddress + ((ulong)depthBlock * 2 * sizeof(ulong)), result); + // Query reducers can read a GPU-cached copy of this range. Use the same + // ordered transfer path as clears instead of changing only CPU backing. + var destination = eventAddress + ((ulong)depthBlock * 2 * sizeof(ulong)); + _host.FillBuffer(destination, sizeof(uint), (uint)result, false); + _host.FillBuffer(destination + sizeof(uint), sizeof(uint), (uint)(result >> 32), false); } - SyntheticOcclusionCounter = (SyntheticOcclusionCounter + 1) & (readyBit - 1); + SyntheticOcclusionCounter = (SyntheticOcclusionCounter + visibleCounterStep) & (readyBit - 1); break; } diff --git a/src/SharpEmu.Libs/Gpu/GpuCommands/GpuCommandInterpreter.Waits.cs b/src/SharpEmu.Libs/Gpu/GpuCommands/GpuCommandInterpreter.Waits.cs index 7e6f8fd..0883dd6 100644 --- a/src/SharpEmu.Libs/Gpu/GpuCommands/GpuCommandInterpreter.Waits.cs +++ b/src/SharpEmu.Libs/Gpu/GpuCommands/GpuCommandInterpreter.Waits.cs @@ -42,7 +42,10 @@ public sealed partial class GpuCommandInterpreter if (!satisfied) { Suspend(); + return; } + + _host.NoteMemoryVisibilityPoint(); } // The wait bit is PredicationZPassWaitOp: 0 (kWaitForQueryResults) stalls the command diff --git a/src/SharpEmu.Libs/Gpu/GpuCommands/GpuCommandInterpreter.cs b/src/SharpEmu.Libs/Gpu/GpuCommands/GpuCommandInterpreter.cs index ef91e56..64e2e31 100644 --- a/src/SharpEmu.Libs/Gpu/GpuCommands/GpuCommandInterpreter.cs +++ b/src/SharpEmu.Libs/Gpu/GpuCommands/GpuCommandInterpreter.cs @@ -443,6 +443,8 @@ public sealed partial class GpuCommandInterpreter { throw _host.Fatal($"The command stream cannot write guest memory: address=0x{address:X16} size={source.Length}."); } + // Later commands in this stream may read the bytes through device addresses. + _host.NoteCommandProcessorWrite(address, (ulong)source.Length); _lastWriteLength = 0; if (source.Length is sizeof(uint) or sizeof(ulong)) { diff --git a/src/SharpEmu.Libs/Gpu/GpuCommands/ICommandStreamHost.cs b/src/SharpEmu.Libs/Gpu/GpuCommands/ICommandStreamHost.cs index 6196265..86b8cb4 100644 --- a/src/SharpEmu.Libs/Gpu/GpuCommands/ICommandStreamHost.cs +++ b/src/SharpEmu.Libs/Gpu/GpuCommands/ICommandStreamHost.cs @@ -38,7 +38,10 @@ public readonly record struct DrawAutoArguments( uint InstanceCount, uint FirstVertex, uint FirstInstance, - DrawOffsetSource OffsetSource); + DrawOffsetSource OffsetSource, + // Nonzero when the counts are still in guest memory at this address, laid out as + // (vertexCount, instanceCount, firstVertex, firstInstance); the other counts are then 1/0. + ulong IndirectArgumentsAddress = 0); public enum EndOfPipeWriteKind { @@ -104,6 +107,17 @@ public interface ICommandStreamHost void CopyBuffer(ulong destination, ulong source, ulong size, bool destinationIsGds, bool sourceIsGds); + // A satisfied wait packet or a command-processor memory write: CPU writes released by the + // waited-on value, or the written bytes, must become visible to the commands that follow, + // the same as at the start of a submission slice. + void NoteMemoryVisibilityPoint() + { + } + + // A command-processor write of these bytes: only they changed, so a host may keep what it + // already synchronized for the rest of memory. + void NoteCommandProcessorWrite(ulong address, ulong size) => NoteMemoryVisibilityPoint(); + void ReadGds(Span destination, uint wordOffset, uint wordCount); void RecordEndOfPipe(in EndOfPipeWrite write); @@ -136,6 +150,9 @@ public interface ICommandStreamHost // guest memory (DrawIndexedArguments.IndirectArgumentsAddress). bool ResolvesIndirectDrawOnGpu => false; + // The host also draws non-indexed indirect draws from their arguments in guest memory. + bool ResolvesNonIndexedIndirectDrawOnGpu => false; + // Called when a queue reset packet clears the processor. void OnQueueReset(int queueId); diff --git a/src/SharpEmu.Libs/Gpu/GpuCommands/Registers/ContextRegisters.cs b/src/SharpEmu.Libs/Gpu/GpuCommands/Registers/ContextRegisters.cs index 5ad7830..c7be50c 100644 --- a/src/SharpEmu.Libs/Gpu/GpuCommands/Registers/ContextRegisters.cs +++ b/src/SharpEmu.Libs/Gpu/GpuCommands/Registers/ContextRegisters.cs @@ -161,7 +161,11 @@ public struct EnhancedQualityAntialiasingRegisters public bool HighQualityIntersections; public bool IncoherentReads; public bool InterpolateComponentZ; + public bool InterpolateSourceZ; public bool StaticAnchorAssociations; + public bool AlphaToMaskEqaaDisable; + public byte OverrasterizationAmount; + public bool EnablePostZOverrasterization; public static EnhancedQualityAntialiasingRegisters Decode(uint value) => new() { @@ -172,7 +176,11 @@ public struct EnhancedQualityAntialiasingRegisters HighQualityIntersections = RegisterField.Bit(value, 16), IncoherentReads = RegisterField.Bit(value, 17), InterpolateComponentZ = RegisterField.Bit(value, 18), + InterpolateSourceZ = RegisterField.Bit(value, 19), StaticAnchorAssociations = RegisterField.Bit(value, 20), + AlphaToMaskEqaaDisable = RegisterField.Bit(value, 21), + OverrasterizationAmount = (byte)RegisterField.Get(value, 24, 0x7), + EnablePostZOverrasterization = RegisterField.Bit(value, 27), }; } @@ -194,6 +202,7 @@ public struct ColorControlRegisters public struct DepthRenderOverrideRegisters { + public bool ForceShaderDepthOrder; public bool ForceZValid; public bool ForceZDirty; public bool ForceStencilValid; @@ -201,6 +210,7 @@ public struct DepthRenderOverrideRegisters public static DepthRenderOverrideRegisters Decode(uint value) => new() { + ForceShaderDepthOrder = RegisterField.Bit(value, 6), ForceZValid = (value & 0x2000_0000u) != 0, ForceZDirty = (value & 0x0800_0000u) != 0, ForceStencilValid = (value & 0x4000_0000u) != 0, @@ -253,10 +263,11 @@ public struct DepthShaderControlRegisters public bool DualExportEnable; public bool ExecuteOnNoop; public bool AlphaToMaskDisable; + public bool DepthBeforeShader; public static DepthShaderControlRegisters Decode(uint value) => new() { - RemainingBits = value & 0xFFFF_908Eu, + RemainingBits = value & 0xFFFF_808Eu, ConservativeDepthExport = (byte)RegisterField.Get(value, 13, 0x3), DepthExportOrder = (byte)RegisterField.Get(value, 4, 0x3), KillEnable = RegisterField.Bit(value, 6), @@ -265,6 +276,7 @@ public struct DepthShaderControlRegisters DualExportEnable = RegisterField.Bit(value, 9), ExecuteOnNoop = RegisterField.Bit(value, 10), AlphaToMaskDisable = RegisterField.Bit(value, 11), + DepthBeforeShader = RegisterField.Bit(value, 12), }; } @@ -292,6 +304,17 @@ public sealed class SampleLocationRegisters public ulong CentroidPriority; public uint[] Locations = new uint[LocationCount]; + // Four-bit signed offsets per coordinate, relative to the pixel center. + public (float X, float Y) Position(uint pixel, uint sample) + { + ArgumentOutOfRangeException.ThrowIfGreaterThanOrEqual(pixel, 4u); + ArgumentOutOfRangeException.ThrowIfGreaterThanOrEqual(sample, 16u); + var packed = Locations[pixel * 4 + sample / 4] >> (int)((sample % 4) * 8); + var x = (int)(packed << 28) >> 28; + var y = (int)(packed << 24) >> 28; + return ((x + 8) / 16f, (y + 8) / 16f); + } + public SampleLocationRegisters Copy() { var copy = (SampleLocationRegisters)MemberwiseClone(); @@ -366,7 +389,11 @@ public sealed class ContextRegisters public ColorControlRegisters ColorControl = new(); public DepthRenderOverrideRegisters DepthRenderOverride; public ScanModeRegisters ScanMode = new(); + public uint ScanModeControl1; public SampleLocationRegisters SampleLocations = new(); + public uint ShaderSampleExclusionMask; + public uint SampleCoverageMaskX0Y0X1Y0 = uint.MaxValue; + public uint SampleCoverageMaskX0Y1X1Y1 = uint.MaxValue; public AntialiasingConfigRegisters AntialiasingConfig; public uint ShaderStages; public DepthTargetWords DepthTarget; diff --git a/src/SharpEmu.Libs/Gpu/GpuCommands/Registers/RegisterOffsets.cs b/src/SharpEmu.Libs/Gpu/GpuCommands/Registers/RegisterOffsets.cs index ef96974..17c71ea 100644 --- a/src/SharpEmu.Libs/Gpu/GpuCommands/Registers/RegisterOffsets.cs +++ b/src/SharpEmu.Libs/Gpu/GpuCommands/Registers/RegisterOffsets.cs @@ -205,6 +205,9 @@ public static class ShaderRegisterOffset public const uint SpiShaderUserAccumLsHs0 = 0x132; public const uint SpiShaderPgmLoLs = 0x148; public const uint SpiShaderPgmHiLs = 0x149; + // Past the LS block, where gfx10 documents no register. Ghost of Yotei 2.002 writes 1 here + // from a shader register table at startup; no shader state depends on it. + public const uint SpiShaderUndocumented192 = 0x192; public const uint ComputeStartX = 0x204; public const uint ComputeStartY = 0x205; public const uint ComputeStartZ = 0x206; @@ -226,6 +229,9 @@ public static class ShaderRegisterOffset // User-config bank offsets the write table covers. public static class UserConfigRegisterOffset { + // Below the primitive-assembly block, where gfx10 to gfx12 document no register. Ghost of + // Yotei 2.002 writes 0 here from a user-config table, next to shader register 0x192. + public const uint UserConfigUndocumented191 = 0x191; public const uint VgtPrimitiveType = 0x242; public const uint VgtIndexType = 0x243; public const uint VgtObjectId = 0x248; diff --git a/src/SharpEmu.Libs/Gpu/GpuCommands/Registers/RegisterWriters.Context.cs b/src/SharpEmu.Libs/Gpu/GpuCommands/Registers/RegisterWriters.Context.cs index ea46226..651c30c 100644 --- a/src/SharpEmu.Libs/Gpu/GpuCommands/Registers/RegisterWriters.Context.cs +++ b/src/SharpEmu.Libs/Gpu/GpuCommands/Registers/RegisterWriters.Context.cs @@ -21,8 +21,8 @@ internal static partial class RegisterWriters { DbCountControl, DbRenderOverride2, DbDfsmControl, DbRmiL2CacheControl, CbRmiGl2CacheControl, TaBcBaseAddr, TaBcBaseAddrHi, CbDccControl, PaSuPointSize, PaSuPointMinMax, DbAlphaToMask, VgtDrawPayloadCntl, - VgtPrimitiveIdReset, PaClObjPrimIdCntl, PaSuVtxCntl, PaScFovWindowLr, PaScFovWindowTb, PaScModeCntl1, - PaScAaMaskX0Y0X1Y0, PaScAaMaskX0Y1X1Y1, PsShaderSampleExclusionMask, PaScBinnerCntl0, PaScBinnerCntl1, + VgtPrimitiveIdReset, PaClObjPrimIdCntl, PaSuVtxCntl, PaScFovWindowLr, PaScFovWindowTb, + PaScBinnerCntl0, PaScBinnerCntl1, PaScConservativeRasterizationCntl, }) { @@ -62,6 +62,7 @@ internal static partial class RegisterWriters direct[PaClVteCntl] = ViewportTransformControlPacket; direct[PaSuLineCntl] = LineControlPacket; direct[PaScModeCntl0] = ScanModeControlPacket; + direct[PaScModeCntl1] = ScanModeControl1Packet; direct[PaScAaConfig] = AntialiasingConfigPacket; for (var offset = PaScAaSampleLocations0; offset < PaScAaSampleLocations0 + SampleLocationRegisters.LocationCount; offset++) { @@ -168,13 +169,17 @@ internal static partial class RegisterWriters CbDccControl, DbCountControl, DbSResultsCompareState0, DbSResultsCompareState1, DbRenderOverride2, DbDfsmControl, DbRmiL2CacheControl, CbRmiGl2CacheControl, TaBcBaseAddr, TaBcBaseAddrHi, PaSuPointSize, PaSuPointMinMax, SpiTmpringSize, VgtDrawPayloadCntl, VgtPrimitiveIdReset, PaClObjPrimIdCntl, PaScFovWindowLr, PaScFovWindowTb, PaScFsrEnable, - FsrRecursions0, FsrRecursions1, PaScModeCntl1, PaScAaMaskX0Y0X1Y0, PaScAaMaskX0Y1X1Y1, PaSuVtxCntl, PsShaderSampleExclusionMask, + FsrRecursions0, FsrRecursions1, PaSuVtxCntl, PaScBinnerCntl0, PaScBinnerCntl1, PaScConservativeRasterizationCntl, DbAlphaToMask, }) { indirect[offset] = IgnoreEntry; } + indirect[PsShaderSampleExclusionMask] = static (banks, _, value) => banks.Context.ShaderSampleExclusionMask = value; + indirect[PaScAaMaskX0Y0X1Y0] = static (banks, _, value) => banks.Context.SampleCoverageMaskX0Y0X1Y0 = value; + indirect[PaScAaMaskX0Y1X1Y1] = static (banks, _, value) => banks.Context.SampleCoverageMaskX0Y1X1Y1 = value; + indirect[SpiVsOutConfig] = static (banks, _, value) => banks.Context.ShaderInterface.VertexOutputConfiguration = value; indirect[DbRenderOverride] = static (banks, _, value) => banks.Context.DepthRenderOverride = DepthRenderOverrideRegisters.Decode(value); indirect[SpiShaderPosFormat] = static (banks, _, value) => banks.Context.ShaderInterface.PositionExportFormat = value; @@ -250,6 +255,7 @@ internal static partial class RegisterWriters indirect[PaSuLineCntl] = LineControlEntry; indirect[PaClVteCntl] = static (banks, _, value) => banks.Context.ScreenViewport.TransformControl = value; indirect[PaScModeCntl0] = static (banks, _, value) => banks.Context.ScanMode = ScanModeRegisters.Decode(value); + indirect[PaScModeCntl1] = static (banks, _, value) => banks.Context.ScanModeControl1 = value; indirect[PaScAaConfig] = static (banks, _, value) => banks.Context.AntialiasingConfig = AntialiasingConfigRegisters.Decode(value); indirect[PaScCentroidPriority0] = CentroidPriorityEntry; indirect[PaScCentroidPriority1] = CentroidPriorityEntry; @@ -404,6 +410,9 @@ internal static partial class RegisterWriters private static uint ScanModeControlPacket(RegisterBanks banks, in PacketContext packet, uint offset, ReadOnlySpan values) => SingleValue(banks, in packet, offset, values, RegisterWriteTable.ContextIndirect[PaScModeCntl0]!); + private static uint ScanModeControl1Packet(RegisterBanks banks, in PacketContext packet, uint offset, ReadOnlySpan values) => + SingleValue(banks, in packet, offset, values, RegisterWriteTable.ContextIndirect[PaScModeCntl1]!); + private static uint AntialiasingConfigPacket(RegisterBanks banks, in PacketContext packet, uint offset, ReadOnlySpan values) => SingleValue(banks, in packet, offset, values, RegisterWriteTable.ContextIndirect[PaScAaConfig]!); diff --git a/src/SharpEmu.Libs/Gpu/GpuCommands/Registers/RegisterWriters.Shader.cs b/src/SharpEmu.Libs/Gpu/GpuCommands/Registers/RegisterWriters.Shader.cs index 00e8036..d9e577a 100644 --- a/src/SharpEmu.Libs/Gpu/GpuCommands/Registers/RegisterWriters.Shader.cs +++ b/src/SharpEmu.Libs/Gpu/GpuCommands/Registers/RegisterWriters.Shader.cs @@ -23,7 +23,6 @@ internal static partial class RegisterWriters [ SpiShaderPaceIdPs, SpiGraphicsShaderControlPs, SpiShaderPaceIdGs, SpiShaderPgmRsrc4Gs, SpiGraphicsShaderControlGs, SpiShaderUserDataAddrLoGs, SpiShaderUserDataAddrHiGs, SpiShaderPgmChksumHs, SpiShaderPgmRsrc4Hs, SpiGraphicsShaderControlHs, - SpiShaderUserDataAddrLoHs, SpiShaderUserDataAddrHiHs, ]; public static void FillShader(RegisterPacketWriter?[] direct, RegisterWriter?[] indirect) @@ -67,6 +66,8 @@ internal static partial class RegisterWriters indirect[offset] = IgnoreEntry; } + indirect[SpiShaderUndocumented192] = UndocumentedShaderEntry; + direct[SpiGraphicsShaderControlPs] = ForwardShaderPacket; direct[SpiGraphicsShaderControlGs] = ForwardShaderPacket; direct[SpiShaderUserDataAddrLoGs] = ForwardShaderPacket; @@ -75,6 +76,13 @@ internal static partial class RegisterWriters direct[SpiShaderUserDataAddrLoHs] = ForwardShaderPacket; direct[SpiShaderUserDataAddrHiHs] = ForwardShaderPacket; + // Unlike program addresses, these registers contain byte-address + // bits 0..31 and 32..47, consumed directly by merged LS/HS s0:s1. + indirect[SpiShaderUserDataAddrLoHs] = static (banks, _, value) => + banks.Shader.Vertex.HullUserDataAddress = (banks.Shader.Vertex.HullUserDataAddress & 0xFFFF_0000_0000ul) | value; + indirect[SpiShaderUserDataAddrHiHs] = static (banks, _, value) => + banks.Shader.Vertex.HullUserDataAddress = (banks.Shader.Vertex.HullUserDataAddress & uint.MaxValue) | ((ulong)(value & 0xFFFFu) << 32); + indirect[SpiShaderPgmLoHs] = static (banks, _, value) => banks.Shader.Vertex.HullAddress = RegisterField.WithLowAddress(banks.Shader.Vertex.HullAddress, value); indirect[SpiShaderPgmHiHs] = static (banks, _, value) => banks.Shader.Vertex.HullAddress = RegisterField.WithHighAddress(banks.Shader.Vertex.HullAddress, value); indirect[SpiShaderPgmRsrc1Hs] = static (banks, _, value) => banks.Shader.Vertex.HullResource1 = HullResource1.Decode(value); @@ -99,6 +107,17 @@ internal static partial class RegisterWriters indirect[SpiShaderPgmRsrc2Vs] = static (banks, _, value) => banks.Shader.Vertex.LegacyVertexResource2 = value; } + // The value is not applied, but each new one is logged so a rendering difference can be traced to it. + private static long _lastUndocumentedShaderValue = -1; + + private static void UndocumentedShaderEntry(RegisterBanks banks, uint offset, uint value) + { + if (Interlocked.Exchange(ref _lastUndocumentedShaderValue, value) != value) + { + Console.Error.WriteLine($"[GPU][WARN] Undocumented shader register written: offset=0x{offset:X4} value=0x{value:X8}; it is not applied."); + } + } + private static uint PixelUserScalarsPacket(RegisterBanks banks, in PacketContext packet, uint offset, ReadOnlySpan values) => UserScalarsPacket(banks, in packet, offset, values, banks.Shader.Pixel.UserScalars, SpiShaderUserDataPs0, GraphicsUserScalarCount); diff --git a/src/SharpEmu.Libs/Gpu/GpuCommands/Registers/RegisterWriters.UserConfig.cs b/src/SharpEmu.Libs/Gpu/GpuCommands/Registers/RegisterWriters.UserConfig.cs index d41d6be..aeaaf2e 100644 --- a/src/SharpEmu.Libs/Gpu/GpuCommands/Registers/RegisterWriters.UserConfig.cs +++ b/src/SharpEmu.Libs/Gpu/GpuCommands/Registers/RegisterWriters.UserConfig.cs @@ -50,6 +50,17 @@ internal static partial class RegisterWriters indirect[GdsOaCounter] = OrderedAppendEntry; indirect[GdsOaAddress] = OrderedAppendEntry; indirect[GeStereoCntl] = IgnoreEntry; + indirect[UserConfigUndocumented191] = UndocumentedUserConfigEntry; + } + + private static long _lastUndocumentedUserConfigValue = -1; + + private static void UndocumentedUserConfigEntry(RegisterBanks banks, uint offset, uint value) + { + if (Interlocked.Exchange(ref _lastUndocumentedUserConfigValue, value) != value) + { + Console.Error.WriteLine($"[GPU][WARN] Undocumented user-config register written: offset=0x{offset:X4} value=0x{value:X8}; it is not applied."); + } } private static uint PrimitiveTypePacket(RegisterBanks banks, in PacketContext packet, uint offset, ReadOnlySpan values) => diff --git a/src/SharpEmu.Libs/Gpu/GpuCommands/Registers/ShaderProgramRegisters.cs b/src/SharpEmu.Libs/Gpu/GpuCommands/Registers/ShaderProgramRegisters.cs index e2d3e45..fd95208 100644 --- a/src/SharpEmu.Libs/Gpu/GpuCommands/Registers/ShaderProgramRegisters.cs +++ b/src/SharpEmu.Libs/Gpu/GpuCommands/Registers/ShaderProgramRegisters.cs @@ -182,6 +182,7 @@ public sealed class VertexStageRegisters public ulong ExportAddress; public ulong LocalAddress; public ulong HullAddress; + public ulong HullUserDataAddress; public HullResource1 HullResource1; public HullResource2 HullResource2; public ulong GeometryAddress; diff --git a/src/SharpEmu.Libs/Gpu/IGuestGpuBackend.cs b/src/SharpEmu.Libs/Gpu/IGuestGpuBackend.cs index c8b3160..ae4d9fa 100644 --- a/src/SharpEmu.Libs/Gpu/IGuestGpuBackend.cs +++ b/src/SharpEmu.Libs/Gpu/IGuestGpuBackend.cs @@ -28,6 +28,9 @@ internal interface IGuestGpuBackend // Compiles one permutation of a program over its resource plan and binding layout. bool TryCompileProgram(ShaderCompileRequest request, out IGuestCompiledShader? shader, out string error); + (IGuestCompiledShader Vertex, IGuestCompiledShader Control) CompileTessellationBridge( + SharpEmu.ShaderCompiler.Resources.BindingLayout layout, Gen5TessellationDomain domain) => + throw new NotSupportedException($"The {BackendName} backend does not support merged tessellation."); /// Returns the backend's no-color-output fragment shader. IGuestCompiledShader GetDepthOnlyFragmentShader(); diff --git a/src/SharpEmu.Libs/Gpu/Images/CachedImage.Scaling.cs b/src/SharpEmu.Libs/Gpu/Images/CachedImage.Scaling.cs new file mode 100644 index 0000000..c5009c7 --- /dev/null +++ b/src/SharpEmu.Libs/Gpu/Images/CachedImage.Scaling.cs @@ -0,0 +1,135 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.Libs.Gpu.Scheduling; +using Silk.NET.Vulkan; + +namespace SharpEmu.Libs.Gpu.Images; + +// The guest-resolution twin of a scaled image and the blits that bridge the two resolutions. +// +// Guest memory layouts never change. Every path that moves guest bytes, and every copy that +// pairs this image with one at another host resolution, runs against the twin at the guest +// extent and then blits across, so callers keep describing rectangles in guest pixels. +public sealed unsafe partial class CachedImage +{ + private CachedImage? _guestSizedTwin; + + private CachedImage GuestSizedTwin() + { + if (_guestSizedTwin is { } existing) + { + return existing; + } + + // The twin is host-owned: dropping the guest placement keeps the image cache from + // tracking it and keeps RenderScalePolicy from scaling it in turn. + var description = Description; + description.Data = default; + description.Stencil = default; + description.Metadata = default; + description.HtileClearMask = uint.MaxValue; + _guestSizedTwin = new CachedImage(_device, _scheduler, _guestBacking, description); + return _guestSizedTwin; + } + + // The image at guest resolution: this image when it is not scaled, otherwise a twin + // holding a downscaled copy of its current contents. + internal CachedImage AtGuestResolution() + { + if (!IsScaled) + { + return this; + } + + var twin = GuestSizedTwin(); + twin.BlitFrom(this); + return twin; + } + + private void BlitFrom(CachedImage source) + { + var aspect = ViewFormatRules.FullAspects(Backing.Format); + if (source.Backing.ImageType != ImageType.Type2D || Backing.ImageType != ImageType.Type2D || + source.Backing.Samples != 1 || Backing.Samples != 1 || + source.Backing.Format != Backing.Format || (aspect & ImageAspectFlags.StencilBit) != 0) + { + throw SubmissionScheduler.Fatal( + $"A scaled image blit needs single-sample 2D images of one format without stencil: " + + $"sourceFormat={(int)source.Backing.Format} format={(int)Backing.Format} " + + $"sourceType={(int)source.Backing.ImageType} type={(int)Backing.ImageType} " + + $"sourceSamples={source.Backing.Samples} samples={Backing.Samples}."); + } + + _scheduler.EndRendering(); + var command = new CommandBuffer(_scheduler.Current.Handle); + source.Transition(ImageLayout.TransferSrcOptimal, AccessFlags.TransferReadBit, null, command); + Transition(ImageLayout.TransferDstOptimal, AccessFlags.TransferWriteBit, null, command); + var levels = Math.Min(source.Backing.MipLevels, Backing.MipLevels); + var layers = Math.Min(source.Backing.Layers, Backing.Layers); + var regions = new ImageBlit[levels]; + for (uint level = 0; level < levels; level++) + { + var subresource = new ImageSubresourceLayers(aspect, level, 0, layers); + regions[level] = new ImageBlit + { + SrcSubresource = subresource, + DstSubresource = subresource, + SrcOffsets = new ImageBlit.SrcOffsetsBuffer + { + Element1 = new Offset3D( + (int)Math.Max(source.Backing.Extent.Width >> (int)level, 1), + (int)Math.Max(source.Backing.Extent.Height >> (int)level, 1), + 1), + }, + DstOffsets = new ImageBlit.DstOffsetsBuffer + { + Element1 = new Offset3D( + (int)Math.Max(Backing.Extent.Width >> (int)level, 1), + (int)Math.Max(Backing.Extent.Height >> (int)level, 1), + 1), + }, + }; + } + + // Depth values must not be interpolated, and Vulkan forbids a filtered depth blit. + var filter = (aspect & ImageAspectFlags.DepthBit) != 0 ? Filter.Nearest : Filter.Linear; + fixed (ImageBlit* blits = regions) + { + _device.Vk.CmdBlitImage(command, source.Backing.Handle, ImageLayout.TransferSrcOptimal, + Backing.Handle, ImageLayout.TransferDstOptimal, (uint)regions.Length, blits, filter); + } + + Transition(ReadyLayout, ReadyAccess, null, command); + } + + // True when a copy between the two images would cross host resolutions. Host-owned + // helpers are built to match the image they mirror, so only guest-placed pairs qualify. + private bool CrossesRenderScale(CachedImage other) => + RenderScale != other.RenderScale && IsGuestPlaced && other.IsGuestPlaced; + + // Runs the copy at guest resolution: the scaled side contributes or receives its twin, + // so the rectangles the caller describes keep their guest meaning. + private bool TryCopyAcrossScales(CachedImage source, Action copy) + { + if (!CrossesRenderScale(source)) + { + return false; + } + + var guestSource = source.AtGuestResolution(); + if (!IsScaled) + { + copy(this, guestSource); + return true; + } + + // The copy may touch only part of the image, so the twin starts from the current + // contents and the result is blitted back over the whole image. + var twin = GuestSizedTwin(); + twin.BlitFrom(this); + copy(twin, guestSource); + BlitFrom(twin); + return true; + } +} diff --git a/src/SharpEmu.Libs/Gpu/Images/CachedImage.Transfers.cs b/src/SharpEmu.Libs/Gpu/Images/CachedImage.Transfers.cs index f244c77..2575926 100644 --- a/src/SharpEmu.Libs/Gpu/Images/CachedImage.Transfers.cs +++ b/src/SharpEmu.Libs/Gpu/Images/CachedImage.Transfers.cs @@ -14,6 +14,14 @@ public sealed unsafe partial class CachedImage { private const AccessFlags WriteAccess = AccessFlags.TransferWriteBit | AccessFlags.ShaderWriteBit | AccessFlags.MemoryWriteBit; private const AccessFlags TransferAccess = AccessFlags.TransferReadBit | AccessFlags.TransferWriteBit; + private const AccessFlags AnyWriteAccess = WriteAccess | AccessFlags.ColorAttachmentWriteBit | AccessFlags.DepthStencilAttachmentWriteBit; + + // Reads after reads in one layout need no barrier: the state keeps every reader, so the next + // writer still waits for all of them. (A sampled depth attachment followed by a draw that only + // tests depth would otherwise end the rendering scope.) + private static bool IsReadAfterRead(ImageAccessState state, ImageLayout layout, AccessFlags access) => + state.Layout == layout && state.Layout != ImageLayout.Undefined && access != 0 && state.Access != 0 && + (state.Access & AnyWriteAccess) == 0 && (access & AnyWriteAccess) == 0; private const AccessFlags MemoryAccess = AccessFlags.MemoryReadBit | AccessFlags.MemoryWriteBit; private const ImageLayout ReadyLayout = ImageLayout.General; private const AccessFlags ReadyAccess = AccessFlags.ShaderReadBit | AccessFlags.TransferReadBit; @@ -32,9 +40,12 @@ public sealed unsafe partial class CachedImage }; // Barriers for the requested state; a repeated write always gets a barrier. + private static readonly List NoBarriers = new(0); + public (List Barriers, PipelineStageFlags SourceStages) GetBarriers(ImageLayout layout, AccessFlags access, PipelineStageFlags stage, SubresourceRange? range) { - var barriers = new List(); + // Most requests need no barrier; the list exists only once one does. Callers never add to it. + List? barriers = null; PipelineStageFlags sourceStages = 0; if ((access & (WriteAccess | AccessFlags.ColorAttachmentWriteBit | AccessFlags.DepthStencilAttachmentWriteBit)) != 0) { @@ -81,9 +92,13 @@ public sealed unsafe partial class CachedImage var state = states[index]; var repeatedWrite = (state.Access & WriteAccess) != 0; - if (state.Layout != layout || state.Access != access || repeatedWrite) + if (IsReadAfterRead(state, layout, access)) { - barriers.Add(MakeBarrier(state, layout, access, level, 1, layer, 1)); + states[index] = new ImageAccessState(state.Stage | stage, state.Access | access, layout); + } + else if (state.Layout != layout || state.Access != access || repeatedWrite) + { + (barriers ??= new List(1)).Add(MakeBarrier(state, layout, access, level, 1, layer, 1)); sourceStages |= state.Stage; states[index] = new ImageAccessState(stage, access, layout); } @@ -101,15 +116,54 @@ public sealed unsafe partial class CachedImage var repeatedWrite = (state.Access & WriteAccess) != 0; if (state.Layout == layout && state.Access == access && !repeatedWrite) { - return (barriers, sourceStages); + return (barriers ?? NoBarriers, sourceStages); } - barriers.Add(MakeBarrier(state, layout, access, 0, Vk.RemainingMipLevels, 0, Vk.RemainingArrayLayers)); + if (IsReadAfterRead(state, layout, access)) + { + Backing.State = new ImageAccessState(state.Stage | stage, state.Access | access, layout); + return (barriers ?? NoBarriers, sourceStages); + } + + (barriers ??= new List(1)).Add(MakeBarrier(state, layout, access, 0, Vk.RemainingMipLevels, 0, Vk.RemainingArrayLayers)); sourceStages |= state.Stage; } Backing.State = new ImageAccessState(stage, access, layout); - return (barriers, sourceStages); + return (barriers ?? NoBarriers, sourceStages); + } + + // The layout every subresource in the range is in, or null when they differ. + public ImageLayout? UniformLayout(SubresourceRange range) + { + var states = Backing.SubresourceStates; + if (states == null) + { + return Backing.State.Layout; + } + + var layers = Description.Resources.Layers; + if (Description.IsVolume) + { + range = range with { BaseLayer = 0, LayerCount = 1 }; + } + + ImageLayout? layout = null; + for (var level = range.BaseLevel; level < range.BaseLevel + range.LevelCount; level++) + { + for (var layer = range.BaseLayer; layer < range.BaseLayer + range.LayerCount; layer++) + { + var index = (int)(level * layers + layer); + if (index >= states.Count || (layout is { } known && known != states[index].Layout)) + { + return null; + } + + layout = states[index].Layout; + } + } + + return layout; } public void Transition(ImageLayout layout, AccessFlags access, SubresourceRange? range, CommandBuffer command) @@ -134,7 +188,7 @@ public sealed unsafe partial class CachedImage // Attachment accesses inside one rendering scope are already ordered by rasterization // order, so a barrier that only repeats an attachment write in the same layout matters // only to what runs after the scope; it waits for the scope to end instead of ending it. - if (OnlyRepeatsAttachmentAccess(barriers) && + if (Backing.SampleLocations is null && OnlyRepeatsAttachmentAccess(barriers) && _scheduler.TryDeferUntilRenderingEnds(sourceStages == 0 ? PipelineStageFlags.TopOfPipeBit : sourceStages, stage, barriers)) { return; @@ -166,9 +220,24 @@ public sealed unsafe partial class CachedImage private void RecordBarriers(CommandBuffer command, PipelineStageFlags sourceStages, PipelineStageFlags destinationStages, BufferMemoryBarrier2* bufferBarrier, List imageBarriers) { - var images = imageBarriers.ToArray(); + // Sample locations chain a stack structure into each barrier, so those barriers are a + // copy; the caller's list never keeps that pointer. + Span images = Backing.SampleLocations is null + ? System.Runtime.InteropServices.CollectionsMarshal.AsSpan(imageBarriers) + : imageBarriers.ToArray(); + fixed (SampleLocationEXT* locations = Backing.SampleLocations) fixed (ImageMemoryBarrier2* imagePointer = images) { + var locationInfo = new SampleLocationsInfoEXT + { + SType = StructureType.SampleLocationsInfoExt, + SampleLocationsPerPixel = SampleCountFlags.Count2Bit, + SampleLocationGridSize = new Extent2D(2, 2), + SampleLocationsCount = 8, + PSampleLocations = locations, + }; + if (locations != null) + for (var index = 0; index < images.Length; index++) imagePointer[index].PNext = &locationInfo; VulkanSynchronization.PipelineBarrier(_device.Vk, command, sourceStages == 0 ? PipelineStageFlags.TopOfPipeBit : sourceStages, destinationStages, DependencyFlags.ByRegionBit, 0, null, bufferBarrier == null ? 0u : 1u, bufferBarrier, (uint)images.Length, imagePointer); @@ -242,6 +311,16 @@ public sealed unsafe partial class CachedImage public void UploadFromBuffer(ReadOnlySpan copies, VkBuffer buffer, ulong offset, ulong size) { + // Guest bytes describe guest geometry, so a scaled image receives them through its + // guest-resolution twin and takes the result as a blit. + if (IsScaled) + { + var twin = GuestSizedTwin(); + twin.UploadFromBuffer(copies, buffer, offset, size); + BlitFrom(twin); + return; + } + var command = BeginTransfer(copies, buffer, size); var sanitized = SanitizeUploadCopies(copies); var uploadCopies = sanitized is null ? copies : sanitized.AsSpan(); @@ -260,6 +339,12 @@ public sealed unsafe partial class CachedImage public void DownloadToBuffer(ReadOnlySpan copies, VkBuffer buffer, ulong offset, ulong size) { + if (IsScaled) + { + AtGuestResolution().DownloadToBuffer(copies, buffer, offset, size); + return; + } + var command = BeginTransfer(copies, buffer, size); var bufferBarrier = BufferBarrier(buffer, offset, size, MemoryAccess, AccessFlags.TransferWriteBit); var (imageBarriers, sourceStages) = GetBarriers(ImageLayout.TransferSrcOptimal, AccessFlags.TransferReadBit, PipelineStageFlags.TransferBit, null); @@ -298,6 +383,11 @@ public sealed unsafe partial class CachedImage // Copies every shared mip level of the source; stencil is never copied here. public void CopyFrom(CachedImage source) { + if (TryCopyAcrossScales(source, static (destination, origin) => destination.CopyFrom(origin))) + { + return; + } + if (source.Backing.Samples != Backing.Samples) { throw SubmissionScheduler.Fatal($"An image copy needs equal sample counts: source={source.Backing.Samples} destination={Backing.Samples}."); @@ -367,6 +457,13 @@ public sealed unsafe partial class CachedImage // Resolves or copies one mip of the source into one mip of this single-sample image. public void ResolveFrom(CachedImage source, in SubresourceRange sourceRange, in SubresourceRange destinationRange) { + var fromRange = sourceRange; + var toRange = destinationRange; + if (TryCopyAcrossScales(source, (destination, origin) => destination.ResolveFrom(origin, fromRange, toRange))) + { + return; + } + if (Backing.Samples != 1 || source.Backing.ImageType != ImageType.Type2D || Backing.ImageType != ImageType.Type2D || sourceRange.LevelCount != 1 || destinationRange.LevelCount != 1 || sourceRange.BaseLevel >= source.Backing.MipLevels || destinationRange.BaseLevel >= Backing.MipLevels || @@ -383,16 +480,18 @@ public sealed unsafe partial class CachedImage var destinationHeight = Math.Max(Backing.Extent.Height >> (int)destinationRange.BaseLevel, 1); var copy = source.Backing.Samples == 1; var formatsAgree = copy ? ViewFormatRules.AreCompatible(source.Backing.Format, Backing.Format) : source.Backing.Format == Backing.Format; - if (layers == 0 || Description.Extent.Width > sourceWidth || Description.Extent.Height > sourceHeight || - Description.Extent.Width > destinationWidth || Description.Extent.Height > destinationHeight || !formatsAgree) + // The resolve moves host texels, so the guest extent crosses into this image's resolution. + var resolveExtent = RenderScalePolicy.ScaleExtent(Description.Extent, RenderScale); + if (layers == 0 || resolveExtent.Width > sourceWidth || resolveExtent.Height > sourceHeight || + resolveExtent.Width > destinationWidth || resolveExtent.Height > destinationHeight || !formatsAgree) { throw SubmissionScheduler.Fatal( - $"The resolve extent or formats do not agree: extent={Description.Extent.Width}x{Description.Extent.Height} source={sourceWidth}x{sourceHeight} destination={destinationWidth}x{destinationHeight} layers={layers} sourceFormat={(int)source.Backing.Format} destinationFormat={(int)Backing.Format}."); + $"The resolve extent or formats do not agree: extent={resolveExtent.Width}x{resolveExtent.Height} source={sourceWidth}x{sourceHeight} destination={destinationWidth}x{destinationHeight} layers={layers} sourceFormat={(int)source.Backing.Format} destinationFormat={(int)Backing.Format}."); } var resolvedSource = sourceRange with { LayerCount = layers }; var resolvedDestination = destinationRange with { LayerCount = layers }; - var extent = new Extent3D(Description.Extent.Width, Description.Extent.Height, 1); + var extent = new Extent3D(resolveExtent.Width, resolveExtent.Height, 1); _scheduler.EndRendering(); var command = new CommandBuffer(_scheduler.Current.Handle); source.Transition(ImageLayout.TransferSrcOptimal, AccessFlags.TransferReadBit, resolvedSource, command); @@ -413,6 +512,13 @@ public sealed unsafe partial class CachedImage public void CopyDepthStencilFrom(CachedImage source, in SubresourceRange range, in Extent3D extent, ImageAspectFlags aspects) { + var copyRange = range; + var copyExtent = extent; + if (TryCopyAcrossScales(source, (destination, origin) => destination.CopyDepthStencilFrom(origin, copyRange, copyExtent, aspects))) + { + return; + } + var requested = aspects & (ImageAspectFlags.DepthBit | ImageAspectFlags.StencilBit); var available = ViewFormatRules.FullAspects(Backing.Format); if (requested == 0 || (requested & ~available) != 0 || source.Backing.Format != Backing.Format || @@ -468,22 +574,27 @@ public sealed unsafe partial class CachedImage } // A storage image holds the stencil bytes while the shader runs; the attachment stays the owner. - public CachedImage CreateStencilStorageImage() + // `width` x `height` is the storage view the shader binds, which can cover only the top-left + // part of the attachment (a dynamic-resolution pass): the shader then sees that size. + public CachedImage CreateStencilStorageImage(uint width, uint height) { if ((ViewFormatRules.FullAspects(Backing.Format) & ImageAspectFlags.StencilBit) == 0 || - Backing.ImageType != ImageType.Type2D || Backing.Samples != 1 || Backing.MipLevels != 1) + Backing.ImageType != ImageType.Type2D || Backing.Samples != 1 || Backing.MipLevels != 1 || + width == 0 || height == 0 || width > Backing.Extent.Width || height > Backing.Extent.Height) { - throw SubmissionScheduler.Fatal("Stencil storage needs a single-sample, single-level 2D stencil image."); + throw SubmissionScheduler.Fatal( + $"Stencil storage needs a single-sample, single-level 2D stencil image covering the view: " + + $"view={width}x{height} attachment={Backing.Extent.Width}x{Backing.Extent.Height} mips={Backing.MipLevels} samples={Backing.Samples}."); } var description = ImageDescription.Create(); description.PixelFormat = Format.R8Uint; description.GuestFormat = GuestPixelFormat.Bits8UInt; - description.Extent = Backing.Extent; + description.Extent = new Extent3D(width, height, 1); description.Resources = new SubresourceCount(1, Backing.Layers); - description.Pitch = Backing.Extent.Width; + description.Pitch = width; description.BytesPerBlock = 1; - return new CachedImage(_device, _scheduler, _guestBacking, description); + return new CachedImage(_device, _scheduler, _guestBacking, description, memoryPool: _memoryPool); } public void CopyStencilStorage(CachedImage storage, GpuBuffer buffer, bool writeBack) @@ -492,11 +603,12 @@ public sealed unsafe partial class CachedImage Backing.ImageType != ImageType.Type2D || Backing.Samples != 1 || Backing.MipLevels != 1 || storage.Backing.Format != Format.R8Uint || storage.Backing.ImageType != ImageType.Type2D || storage.Backing.Samples != 1 || storage.Backing.MipLevels != 1 || storage.Backing.Layers != Backing.Layers || - storage.Backing.Extent.Width != Backing.Extent.Width || storage.Backing.Extent.Height != Backing.Extent.Height) + storage.Backing.Extent.Width > Backing.Extent.Width || storage.Backing.Extent.Height > Backing.Extent.Height) { throw SubmissionScheduler.Fatal("The stencil storage image does not match its attachment."); } + // Only the storage's region moves: the rest of the plane keeps its stencil values. if (writeBack) CopyThroughBuffer(storage, buffer, ImageAspectFlags.ColorBit, ImageAspectFlags.StencilBit); else @@ -510,6 +622,11 @@ public sealed unsafe partial class CachedImage private void CopyThroughBuffer(CachedImage source, GpuBuffer buffer, ImageAspectFlags sourceAspect, ImageAspectFlags destinationAspect) { + if (TryCopyAcrossScales(source, (destination, origin) => destination.CopyThroughBuffer(origin, buffer, sourceAspect, destinationAspect))) + { + return; + } + if (buffer.Handle.Handle == 0 || source.Backing.Samples != 1 || Backing.Samples != 1) { throw SubmissionScheduler.Fatal($"A copy through a buffer needs single-sample images and a buffer: sourceSamples={source.Backing.Samples} destinationSamples={Backing.Samples}."); @@ -531,10 +648,11 @@ public sealed unsafe partial class CachedImage var command = new CommandBuffer(_scheduler.Current.Handle); source.Transition(ImageLayout.TransferSrcOptimal, AccessFlags.TransferReadBit, null, command); Transition(ImageLayout.TransferDstOptimal, AccessFlags.TransferWriteBit, null, command); + // The copy covers the region both images have, from their top-left corner. for (uint level = 0; level < levels; level++) { - var width = Math.Max(source.Backing.Extent.Width >> (int)level, 1); - var height = Math.Max(source.Backing.Extent.Height >> (int)level, 1); + var width = Math.Max(Math.Min(source.Backing.Extent.Width, Backing.Extent.Width) >> (int)level, 1); + var height = Math.Max(Math.Min(source.Backing.Extent.Height, Backing.Extent.Height) >> (int)level, 1); var sourceDepth = source.Backing.ImageType == ImageType.Type3D ? Math.Max(source.Backing.Extent.Depth >> (int)level, 1) : source.Backing.Layers; var destinationDepth = Backing.ImageType == ImageType.Type3D ? Math.Max(Backing.Extent.Depth >> (int)level, 1) : Backing.Layers; var slices = Math.Min(sourceDepth, destinationDepth); @@ -579,6 +697,11 @@ public sealed unsafe partial class CachedImage // Copies a whole standalone image into one mip and layer of this image. public void CopyMipFrom(CachedImage source, uint mip, uint layer) { + if (TryCopyAcrossScales(source, (destination, origin) => destination.CopyMipFrom(origin, mip, layer))) + { + return; + } + if (source.Backing.Samples != Backing.Samples || mip >= Backing.MipLevels || layer >= Backing.Layers) { throw SubmissionScheduler.Fatal($"The mip copy target is invalid: mip={mip} layer={layer} levels={Backing.MipLevels} layers={Backing.Layers} sourceSamples={source.Backing.Samples} destinationSamples={Backing.Samples}."); @@ -627,6 +750,11 @@ public sealed unsafe partial class CachedImage // Copies a rectangle of the source's first mip into one mip of this image, at its origin. public void CopyRegionFrom(CachedImage source, uint sourceX, uint sourceY, uint mip, uint width, uint height) { + if (TryCopyAcrossScales(source, (destination, origin) => destination.CopyRegionFrom(origin, sourceX, sourceY, mip, width, height))) + { + return; + } + if (mip >= Backing.MipLevels || sourceX + width > source.Backing.Extent.Width || sourceY + height > source.Backing.Extent.Height) { throw SubmissionScheduler.Fatal( diff --git a/src/SharpEmu.Libs/Gpu/Images/CachedImage.cs b/src/SharpEmu.Libs/Gpu/Images/CachedImage.cs index 365bced..6a492b1 100644 --- a/src/SharpEmu.Libs/Gpu/Images/CachedImage.cs +++ b/src/SharpEmu.Libs/Gpu/Images/CachedImage.cs @@ -49,6 +49,7 @@ public sealed class ImageBacking public ImageCreateFlags Flags; public Image Handle; public ImageAccessState State = ImageAccessState.Initial; + public SampleLocationEXT[]? SampleLocations; public List? SubresourceStates; public DeviceMemory Memory; public ulong AllocationSize; @@ -63,6 +64,9 @@ public sealed unsafe partial class CachedImage : IDisposable private readonly SubmissionScheduler _scheduler; private readonly IGuestBackedSpace _guestBacking; private readonly ImageBackingPool? _pool; + private readonly OptimalImageMemoryPool? _memoryPool; + private readonly Func? _memoryDiagnostics; + private OptimalImageMemoryAllocation? _memoryPlacement; private ImageBackingPool.Key _poolKey; private ulong _maybeCpuHash; private bool _cpuDirty; @@ -73,8 +77,14 @@ public sealed unsafe partial class CachedImage : IDisposable private bool _bufferHoldsGpuContents; public ImageDescription Description; + + // The host resolution multiplier of this image. Guest geometry in Description never + // changes; only the backing is larger or smaller, and guest bytes move through a + // guest-resolution twin (CachedImage.Transfers). + public readonly float RenderScale = 1.0f; public readonly ImageBacking Backing = new(); public readonly List Views = new(); + private readonly Dictionary<(uint Width, uint Height), CachedImage> _stencilStorageImages = new(); public ImageUses Uses; public ImageBindingState Binding; public bool Registered; @@ -86,14 +96,18 @@ public sealed unsafe partial class CachedImage : IDisposable public int RecencyEntryIndex; public CachedImage(GpuDeviceInfo device, SubmissionScheduler scheduler, IGuestBackedSpace guestBacking, in ImageDescription description, - ImageBackingPool? pool = null) + ImageBackingPool? pool = null, OptimalImageMemoryPool? memoryPool = null, Func? memoryDiagnostics = null, + bool allowScaling = true) { _device = device; _scheduler = scheduler; _guestBacking = guestBacking; _pool = pool; + _memoryPool = memoryPool; + _memoryDiagnostics = memoryDiagnostics; Description = description; Description.Validate(); + RenderScale = allowScaling ? RenderScalePolicy.ScaleFor(Description) : 1.0f; _cpuDirty = !ImageDescription.IsEmptyRange(Description.Data) && Description.Metadata.Compression == DisplayCompression.Uncompressed; if (Description.PixelFormat == Format.Undefined) { @@ -102,7 +116,13 @@ public sealed unsafe partial class CachedImage : IDisposable Backing.Format = Description.PixelFormat; Backing.ImageType = HostImageType(Description.Type); - Backing.Extent = Description.Extent; + var guestExtent = Description.FirstLevel == 0 + ? Description.Extent + : new Extent3D( + Math.Max(Description.Extent.Width >> (int)Description.FirstLevel, 1u), + Math.Max(Description.Extent.Height >> (int)Description.FirstLevel, 1u), + Description.Extent.Depth); + Backing.Extent = RenderScalePolicy.ScaleExtent(guestExtent, RenderScale); Backing.GuestPitch = Description.Pitch; Backing.Layers = Description.IsVolume ? 1 : Description.Resources.Layers; Backing.MipLevels = Description.Resources.Levels; @@ -135,7 +155,7 @@ public sealed unsafe partial class CachedImage : IDisposable Backing.Usage = create.Usage; _poolKey = ImageBackingPool.KeyOf(create); - if (_pool is not null && _pool.TryTake(_poolKey, out Backing.Handle, out Backing.Memory, out Backing.AllocationSize)) + if (_pool is not null && _pool.TryTake(_poolKey, out Backing.Handle, out Backing.Memory, out Backing.AllocationSize, out _memoryPlacement)) { return; } @@ -147,11 +167,26 @@ public sealed unsafe partial class CachedImage : IDisposable throw CreateFailure(create, "vkCreateImage", createResult, 0); } - vk.GetImageMemoryRequirements(device.Device, Backing.Handle, out var requirements); + var dedicated = new MemoryDedicatedRequirements { SType = StructureType.MemoryDedicatedRequirements }; + var imageRequirements = new ImageMemoryRequirementsInfo2 + { + SType = StructureType.ImageMemoryRequirementsInfo2, Image = Backing.Handle, + }; + var requirements2 = new MemoryRequirements2 + { + SType = StructureType.MemoryRequirements2, PNext = &dedicated, + }; + vk.GetImageMemoryRequirements2(device.Device, &imageRequirements, &requirements2); + var requirements = requirements2.MemoryRequirements; + var dedicatedInfo = new MemoryDedicatedAllocateInfo + { + SType = StructureType.MemoryDedicatedAllocateInfo, Image = Backing.Handle, + }; var allocateInfo = new MemoryAllocateInfo { SType = StructureType.MemoryAllocateInfo, AllocationSize = requirements.Size, + PNext = dedicated.RequiresDedicatedAllocation ? &dedicatedInfo : null, }; var allocated = Result.ErrorOutOfDeviceMemory; for (uint index = 0; index < device.MemoryTypeCount; index++) @@ -163,7 +198,16 @@ public sealed unsafe partial class CachedImage : IDisposable } allocateInfo.MemoryTypeIndex = index; - allocated = device.AllocateMemory(allocateInfo, out Backing.Memory); + if (_memoryPool is not null && !dedicated.RequiresDedicatedAllocation) + { + allocated = _memoryPool.Allocate(requirements, index, out var placement); + if (allocated == Result.Success) + { + _memoryPlacement = placement; + Backing.Memory = placement.Memory; + } + } + else allocated = device.AllocateMemory(allocateInfo, out Backing.Memory); if (allocated == Result.Success) { break; @@ -173,17 +217,20 @@ public sealed unsafe partial class CachedImage : IDisposable if (allocated != Result.Success) { vk.DestroyImage(device.Device, Backing.Handle, null); - device.FreeMemory(Backing.Memory); + ReleaseMemory(); Backing.Handle = default; Backing.Memory = default; + // Running out of memory is recoverable: the cache can free images and try again. + if (allocated is Result.ErrorOutOfDeviceMemory or Result.ErrorOutOfHostMemory) + throw new ImageOutOfMemoryException(CreateFailureMessage(create, "vkAllocateMemory", allocated, requirements.Size)); throw CreateFailure(create, "vkAllocateMemory", allocated, requirements.Size); } - var bindResult = vk.BindImageMemory(device.Device, Backing.Handle, Backing.Memory, 0); + var bindResult = vk.BindImageMemory(device.Device, Backing.Handle, Backing.Memory, _memoryPlacement?.Offset ?? 0); if (bindResult != Result.Success) { vk.DestroyImage(device.Device, Backing.Handle, null); - device.FreeMemory(Backing.Memory); + ReleaseMemory(); Backing.Handle = default; Backing.Memory = default; throw CreateFailure(create, "vkBindImageMemory", bindResult, requirements.Size); @@ -192,11 +239,27 @@ public sealed unsafe partial class CachedImage : IDisposable Backing.AllocationSize = requirements.Size; } - private static Exception CreateFailure(in ImageCreateInfo create, string operation, Result result, ulong requiredBytes) => - SubmissionScheduler.Fatal( + private void ReleaseMemory() + { + if (_memoryPlacement is { } placement) _memoryPool!.Free(placement); + else _device.FreeMemory(Backing.Memory); + _memoryPlacement = null; + } + + private Exception CreateFailure(in ImageCreateInfo create, string operation, Result result, ulong requiredBytes) => + SubmissionScheduler.Fatal(CreateFailureMessage(create, operation, result, requiredBytes)); + + private string CreateFailureMessage(in ImageCreateInfo create, string operation, Result result, ulong requiredBytes) => $"The image could not be created: operation={operation} result={result} required_bytes={requiredBytes} " + $"extent={create.Extent.Width}x{create.Extent.Height}x{create.Extent.Depth} format={create.Format}({(int)create.Format}) " + - $"layers={create.ArrayLayers} levels={create.MipLevels} usage=0x{(uint)create.Usage:X} flags=0x{(uint)create.Flags:X}."); + $"layers={create.ArrayLayers} levels={create.MipLevels} usage=0x{(uint)create.Usage:X} flags=0x{(uint)create.Flags:X} " + + $"live_allocations={_device.LiveAllocations} allocation_limit={_device.MaxMemoryAllocationCount} " + + $"image_pool_allocated={_memoryPool?.AllocatedBytes ?? 0} image_pool_placed={_memoryPool?.PlacedBytes ?? 0} " + + $"last_failed_allocation_bytes={_device.LastFailedAllocationBytes} guest_address=0x{Description.Data.Address:X16} " + + (_memoryDiagnostics?.Invoke() ?? string.Empty); + + // The device ran out of memory for the image's backing. Nothing was left allocated. + internal sealed class ImageOutOfMemoryException(string message) : Exception(message); internal static bool TrySelectSupportedImageConfiguration(IImageFormatSupport device, ref ImageCreateInfo configuration, bool allowCompressedImageFallback) { @@ -251,9 +314,12 @@ public sealed unsafe partial class CachedImage : IDisposable _ => throw SubmissionScheduler.Fatal($"The image type is not a base type: type={(uint)type}."), }; - private static ImageCreateFlags CreateFlags(in ImageDescription description) + private ImageCreateFlags CreateFlags(in ImageDescription description) { ImageCreateFlags flags = 0; + if (_device.CustomTwoSampleLocationsSupported && description.Samples == 2 && + DepthFormatRule.AspectTransferFormat(description.PixelFormat) != Format.Undefined) + flags |= ImageCreateFlags.CreateSampleLocationsCompatibleDepthBitExt; if (DepthFormatRule.AspectTransferFormat(description.PixelFormat) == Format.Undefined) { flags |= ImageCreateFlags.CreateMutableFormatBit | ImageCreateFlags.CreateExtendedUsageBit; @@ -320,6 +386,13 @@ public sealed unsafe partial class CachedImage : IDisposable return usage; } + public bool IsScaled => RenderScale != 1.0f; + + // Host-owned helpers - feedback snapshots, the stencil proxy, the guest-resolution twin - + // carry no guest placement and are created to match the image they mirror, so a copy + // between them and that image never crosses a resolution. + internal bool IsGuestPlaced => !ImageDescription.IsEmptyRange(Description.Data); + public void AssociateDepth(ResourceSlotIdentifier depthImage) => DepthOwner = depthImage; internal ulong LastCpuWriteAddress { get; private set; } @@ -427,10 +500,29 @@ public sealed unsafe partial class CachedImage : IDisposable _gpuModified = true; _bufferHoldsGpuContents = false; GpuWriteSequence = Interlocked.Increment(ref _gpuWriteCounter); + ContentSequence = GpuWriteSequence; _guestPieceHashes = null; } - public void ClearGpuModified() => _gpuModified = false; + // The GPU write the image contents reflect. A copy from a same-layout alias takes the + // source's sequence, so the alias is not copied back while neither image is written again. + public long ContentSequence { get; set; } + + public void ClearGpuModified( + [System.Runtime.CompilerServices.CallerFilePath] string file = "", + [System.Runtime.CompilerServices.CallerLineNumber] int line = 0) + { + if (_gpuModified && ImageDropTrace.Enabled) + { + ImageDropTrace.Record(this, file, line); + } + + _gpuModified = false; + } + + // Advances whenever any image becomes GPU-modified, the only way an image starts to own + // guest bytes the CPU must not read. + public static long GpuWriteVersion => Interlocked.Read(ref _gpuWriteCounter); public bool IsBufferModified => _bufferModified; @@ -453,7 +545,38 @@ public sealed unsafe partial class CachedImage : IDisposable public bool IsWatched => WatchBegin != 0 && WatchEnd != 0; - public ulong AccountedSize => Backing.Exists ? (Description.Data.Size + 1023) & ~1023UL : 0; + // Account the device allocation, not the guest byte range. Optimal-tiled images can + // reserve more memory than their guest payload because of tiling, mip alignment and + // implementation-specific requirements. The backing allocation is the quantity the + // image memory pool actually reserves and releases. + public ulong AccountedSize + { + get + { + var bytes = Backing.Exists ? Backing.AllocationSize : 0; + foreach (var image in _stencilStorageImages.Values) bytes += image.AccountedSize; + return bytes; + } + } + + // Queue order permits reuse: every preparation first refreshes the working + // image from the attachment and publishes its writes before the next use. + public CachedImage GetOrCreateStencilStorageImage(uint width, uint height) + { + if (!_stencilStorageImages.TryGetValue((width, height), out var image)) + { + image = CreateStencilStorageImage(width, height); + _stencilStorageImages.Add((width, height), image); + } + return image; + } + + public IEnumerable GetOwnedViews() + { + foreach (var cached in Views) yield return cached.View; + foreach (var storage in _stencilStorageImages.Values) + foreach (var view in storage.GetOwnedViews()) yield return view; + } // Hashes the first and last partial tracker page of the guest data through the backing alias. public ulong HashGuestEdges() @@ -604,6 +727,10 @@ public sealed unsafe partial class CachedImage : IDisposable // Immediate destruction; the owner waits for GPU work before disposing. public void Dispose() { + foreach (var storage in _stencilStorageImages.Values) storage.Dispose(); + _stencilStorageImages.Clear(); + _guestSizedTwin?.Dispose(); + _guestSizedTwin = null; foreach (var cached in Views) { _device.Vk.DestroyImageView(_device.Device, cached.View, null); @@ -612,14 +739,15 @@ public sealed unsafe partial class CachedImage : IDisposable Views.Clear(); if (Backing.Exists) { - if (_pool is null || !_pool.TryReturn(_poolKey, Backing.Handle, Backing.Memory, Backing.AllocationSize)) + if (_pool is null || !_pool.TryReturn(_poolKey, Backing.Handle, Backing.Memory, Backing.AllocationSize, _memoryPlacement, _memoryPool)) { _device.Vk.DestroyImage(_device.Device, Backing.Handle, null); - _device.FreeMemory(Backing.Memory); + ReleaseMemory(); } Backing.Handle = default; Backing.Memory = default; + _memoryPlacement = null; } } } diff --git a/src/SharpEmu.Libs/Gpu/Images/GpuTiler.cs b/src/SharpEmu.Libs/Gpu/Images/GpuTiler.cs index 72a42cd..ba6c89d 100644 --- a/src/SharpEmu.Libs/Gpu/Images/GpuTiler.cs +++ b/src/SharpEmu.Libs/Gpu/Images/GpuTiler.cs @@ -91,6 +91,7 @@ public sealed unsafe class GpuTiler : IDisposable private Pipeline _d24ToD16; private Pipeline _d32ToD16; private Pipeline _swapBgra16; + private readonly Pipeline[] _narrowColorConversions = new Pipeline[4]; public GpuTiler(GpuDeviceInfo device, SubmissionScheduler scheduler, GpuRingBuffer stream) { @@ -146,10 +147,14 @@ public sealed unsafe class GpuTiler : IDisposable DestroyPipeline(_d24ToD16); DestroyPipeline(_d32ToD16); DestroyPipeline(_swapBgra16); + foreach (var pipeline in _narrowColorConversions) + DestroyPipeline(pipeline); _pools.Dispose(); lock (_scratchGate) { _scratchDisposed = true; + _uploadRing?.Dispose(); + _uploadRing = null; foreach (var pooled in _scratchPool.Values) { foreach (var buffer in pooled) @@ -197,6 +202,27 @@ public sealed unsafe class GpuTiler : IDisposable private readonly Dictionary> _scratchPool = new(); private ulong _scratchPooledBytes; + // Scratches only return when their tick completes, so one long tick full of uploads + // (a streaming burst) holds all of them; past this budget the next draw ends the tick. + internal ulong OutstandingScratchBudget { get; set; } = 256UL << 20; + private ulong _outstandingScratch; + + public bool ScratchOverBudget => Interlocked.Read(ref _outstandingScratch) > OutstandingScratchBudget; + internal ulong OutstandingScratchBytes => Interlocked.Read(ref _outstandingScratch); + internal ulong PooledScratchBytes { get { lock (_scratchGate) return _scratchPooledBytes; } } + + public void ReleaseUnusedScratch() + { + lock (_scratchGate) + { + // Only completion callbacks return these buffers. Pending GPU uses + // remain outside the pool, so trimming cannot invalidate them. + foreach (var pooled in _scratchPool.Values) + while (pooled.TryPop(out var buffer)) buffer.Dispose(); + _scratchPooledBytes = 0; + } + } + // The scratch stays alive through the current tick; the caller sees exactly `size` bytes. private TilerBufferSpan AllocateScratch(ulong size) { @@ -219,10 +245,53 @@ public sealed unsafe class GpuTiler : IDisposable buffer ??= new GpuBuffer(_device, _scheduler, GpuBufferUsage.DeviceLocal, 0, BufferUsageFlags.StorageBufferBit | BufferUsageFlags.TransferSrcBit | BufferUsageFlags.TransferDstBit, shift >= 63 ? size : 1UL << shift); - _scheduler.QueueCompletionAction(() => ReturnScratch(shift, buffer)); + var held = buffer.Size; + Interlocked.Add(ref _outstandingScratch, held); + _scheduler.QueueCompletionAction(() => + { + Interlocked.Add(ref _outstandingScratch, unchecked(0UL - held)); + ReturnScratch(shift, buffer); + }); return new TilerBufferSpan(buffer.Handle, 0, size); } + // Upload scratch comes from one ring reused across uploads and ticks. Pooled scratch only + // returns when its tick completes, and a tick cannot end inside a draw, so a bindless + // draw that materializes thousands of textures would hold one scratch per texture. + // Every tile record starts with an all-commands barrier on its target range, which also + // orders it after earlier reads of that range, so reusing the ring stays safe. + private const ulong UploadRingSize = 256UL << 20; + private const ulong UploadRingAlignment = 256; + private GpuBuffer? _uploadRing; + private ulong _uploadRingOffset; + + private TilerBufferSpan AllocateUploadScratch(ulong size) + { + size = (size + 3) & ~3UL; + if (size > UploadRingSize) + { + return AllocateScratch(size); + } + + _uploadRing ??= new GpuBuffer(_device, _scheduler, GpuBufferUsage.DeviceLocal, 0, + BufferUsageFlags.StorageBufferBit | BufferUsageFlags.TransferSrcBit | BufferUsageFlags.TransferDstBit, UploadRingSize); + var alignment = Math.Max(UploadRingAlignment, StorageAlignment); + var offset = (_uploadRingOffset + alignment - 1) / alignment * alignment; + if (offset + size > UploadRingSize) + { + // Wrapping: all earlier ring accesses finish before the ring is written again. + offset = 0; + _scheduler.EndRendering(); + var barrier = Barrier(_uploadRing.Handle, 0, UploadRingSize, + AccessFlags.MemoryReadBit | AccessFlags.MemoryWriteBit, AccessFlags.MemoryReadBit | AccessFlags.MemoryWriteBit); + VulkanSynchronization.PipelineBarrier(_device.Vk, new CommandBuffer(_scheduler.Current.Handle), + PipelineStageFlags.AllCommandsBit, PipelineStageFlags.AllCommandsBit, 0, 0, null, 1, &barrier, 0, null); + } + + _uploadRingOffset = offset + size; + return new TilerBufferSpan(_uploadRing.Handle, offset, size); + } + private void ReturnScratch(int shift, GpuBuffer buffer) { lock (_scratchGate) @@ -601,9 +670,9 @@ public sealed unsafe class GpuTiler : IDisposable var sourceBase = tiledOffset & (StorageAlignment - 1); var dispatches = new List(); Prepare(false, tiledCapacity, linearCapacity, transfers, sourceBase, 0, dispatches); - var scratch = AllocateScratch((linearCapacity + 3) & ~3UL); - Record(false, tiled, tiledOffset, tiledCapacity, scratch.Buffer, 0, scratch.Size, dispatches, true); - return new TilerBufferSpan(scratch.Buffer, 0, linearCapacity); + var scratch = AllocateUploadScratch(linearCapacity); + Record(false, tiled, tiledOffset, tiledCapacity, scratch.Buffer, scratch.Offset, scratch.Size, dispatches, true); + return new TilerBufferSpan(scratch.Buffer, scratch.Offset, linearCapacity); } public void Tile(VkBuffer linear, ulong linearOffset, ulong linearCapacity, VkBuffer tiled, ulong tiledOffset, ulong tiledCapacity, ReadOnlySpan transfers) @@ -709,10 +778,29 @@ public sealed unsafe class GpuTiler : IDisposable pipeline = CreateComputePipeline(spirv, $"depth16 {direction} d32={useFloat32Depth}"); } - var sourceElement = direction == DepthConversionDirection.Widen ? 2UL : 4UL; - var targetElement = direction == DepthConversionDirection.Widen ? 4UL : 2UL; - var sourceActive = layout.Width * sourceElement; - var targetActive = layout.Width * targetElement; + ConvertElements(source, target, pipeline, direction == DepthConversionDirection.Widen ? 2u : 4u, + direction == DepthConversionDirection.Widen ? 4u : 2u, layout); + } + + public void ConvertNarrowColor(TilerBufferSpan source, TilerBufferSpan target, uint channels, bool widen, in DepthConversionLayout layout) + { + if (channels is not (1 or 2)) + throw new ArgumentOutOfRangeException(nameof(channels)); + var wideOffset = widen ? target.Offset : source.Offset; + var wideRow = widen ? layout.TargetRowStride : layout.SourceRowStride; + var wideSlice = widen ? layout.TargetSliceStride : layout.SourceSliceStride; + if (((wideOffset | wideRow | wideSlice) & 3) != 0) + throw SubmissionScheduler.Fatal("The RGBA color conversion storage must be dword aligned."); + ref var pipeline = ref _narrowColorConversions[(channels - 1) * 2 + (widen ? 0 : 1)]; + if (pipeline.Handle == 0) + pipeline = CreateComputePipeline(TilerShaders.CreateNarrowColorConversion(channels, widen), $"narrow color channels={channels} widen={widen}"); + ConvertElements(source, target, pipeline, widen ? channels : 4u, widen ? 4u : channels, layout); + } + + private void ConvertElements(TilerBufferSpan source, TilerBufferSpan target, Pipeline pipeline, uint sourceElement, uint targetElement, in DepthConversionLayout layout) + { + var sourceActive = (ulong)layout.Width * sourceElement; + var targetActive = (ulong)layout.Width * targetElement; if (layout.Width == 0 || layout.SourceRowStride > uint.MaxValue || layout.TargetRowStride > uint.MaxValue) { throw SubmissionScheduler.Fatal($"The depth conversion layout is invalid: width={layout.Width} sourceRowStride={layout.SourceRowStride} targetRowStride={layout.TargetRowStride}."); @@ -845,7 +933,7 @@ public sealed unsafe class GpuTiler : IDisposable throw SubmissionScheduler.Fatal($"The BGRA16 swap input size is invalid: size={input.Size}."); } - var result = AllocateScratch(input.Size); + var result = AllocateUploadScratch(input.Size); SwapBgra16(input, result, (uint)(input.Size / 8)); return result; } diff --git a/src/SharpEmu.Libs/Gpu/Images/GuestImageCache.Collection.cs b/src/SharpEmu.Libs/Gpu/Images/GuestImageCache.Collection.cs index cc71b78..173979f 100644 --- a/src/SharpEmu.Libs/Gpu/Images/GuestImageCache.Collection.cs +++ b/src/SharpEmu.Libs/Gpu/Images/GuestImageCache.Collection.cs @@ -8,6 +8,119 @@ namespace SharpEmu.Libs.Gpu.Images; // Garbage collection by recency and memory pressure, and the scheduled readback flush. public sealed partial class GuestImageCache { + // Cached images allocate their own device memory (see CachedImage), so the pressure + // includes retired images until GPU completion actually releases their memory. + private ulong CollectionMemoryBytes => TotalUsedMemory + + _imageMemoryPool.UnusedBytes + (_backingPool?.PooledBytes ?? 0); + + // Allocation failure runs with the cache lock already held. Enumerate only + // on failure, so these diagnostics add no per-draw walk or logging traffic. + private string DescribeImageMemory() + { + var registered = 0; + ulong cleanBytes = 0, modifiedLinearBytes = 0, modifiedTiledBytes = 0; + _slots.ForEach((_, image) => + { + if (!image.Registered) return; + registered++; + if (!image.IsGpuModified) cleanBytes += image.AccountedSize; + else if (image.Description.IsTiled) modifiedTiledBytes += image.AccountedSize; + else modifiedLinearBytes += image.AccountedSize; + }); + return $"registered_images={registered} clean_image_bytes={cleanBytes} gpu_modified_linear_bytes={modifiedLinearBytes} " + + $"gpu_modified_tiled_bytes={modifiedTiledBytes} retired_image_bytes={Interlocked.Read(ref _retiredImageMemoryBytes)} " + + $"retained_backing_bytes={_backingPool?.PooledBytes ?? 0} collection_bytes={CollectionMemoryBytes} critical_bytes={_criticalMemoryBytes}"; + } + + // Allocation-time pressure is deliberately separate from the periodic sweep: + // a visibility-buffer burst can create thousands of images before the next + // frame boundary. Destruction still goes through DeleteImage's completion + // action, so no image is freed while the current tick can reference it. + private void CollectForAllocation(ulong requiredBytes) + { + FinishRetiredImagesForAllocation(requiredBytes); + // A burst can contain only newly touched images. Once the recency walk finds + // no evictable image, retrying it for every allocation in the same tick is + // pure repeated work; the next tick will make older images eligible. + if (_allocationCollectionBlocked) + { + return; + } + + var current = CollectionMemoryBytes; + if (requiredBytes > _criticalMemoryBytes || current <= _criticalMemoryBytes - requiredBytes) + { + return; + } + + var target = _criticalMemoryBytes - requiredBytes; + var blocked = false; + // Retired images still occupy memory until completion. Do not retire + // the entire old working set merely because that completion is pending. + while (CollectionMemoryBytes - (ulong)Interlocked.Read(ref _retiredImageMemoryBytes) > target) + { + var before = _totalUsedMemory; + Collect(_collectionTick, allowAggressive: true); + if (_totalUsedMemory == before) + { + blocked = true; + break; + } + } + + _allocationCollectionBlocked = blocked; + FinishRetiredImagesForAllocation(requiredBytes); + } + + // An allocation failed despite the budget. Retire every image collection allows, then let + // the retired ones be destroyed and return unused pool memory to the device. + public void ReclaimForAllocation() + { + using var held = _lock.Hold(); + ReclaimAfterFailedAllocation(); + } + + private void ReclaimAfterFailedAllocation() + { + _allocationCollectionBlocked = false; + while (true) + { + var before = _totalUsedMemory; + Collect(_collectionTick, allowAggressive: true); + if (_totalUsedMemory == before) + break; + } + + if (_scheduler.Active && !_scheduler.InsideTickCallback) + { + _lock.Exit(); + try { _scheduler.Finish(); } + finally { _lock.Enter(); } + } + + ReleaseUnusedMemoryCore(); + } + + private void FinishRetiredImagesForAllocation(ulong requiredBytes) + { + var retired = Interlocked.Read(ref _retiredImageMemoryBytes); + if (retired == 0 || !_scheduler.Active || _scheduler.InsideTickCallback) + return; + var resident = CollectionMemoryBytes; + if (!RetirementOverBudget && requiredBytes <= _criticalMemoryBytes && + resident <= _criticalMemoryBytes - requiredBytes) + return; + + // Completion can call back into the cache. Never wait while holding its + // region lock. Images touched by this preparation remain registered; + // only previously retired objects are destroyed by these callbacks. + _lock.Exit(); + try { _scheduler.Finish(); } + finally { _lock.Enter(); } + ReleaseUnusedMemoryCore(); + _allocationCollectionBlocked = false; + } + public void RunGarbageCollector() => RunGarbageCollector(endsFrame: true); // Image ages are counted in frames. Collection also runs after guest submissions, so memory @@ -18,45 +131,84 @@ public sealed partial class GuestImageCache { using var held = _lock.Hold(); var tick = endsFrame ? _collectionTick++ : _collectionTick; - if (_totalUsedMemory < _collectionStartBytes) + _allocationCollectionBlocked = false; + if (MemoryUnderPressure) + ReleaseUnusedMemoryCore(); + else + _imageMemoryPool.ReleaseRetained(); + if (CollectionMemoryBytes < _collectionStartBytes) { return; } - Collect(tick, allowAggressive: false); - if (_totalUsedMemory >= _criticalMemoryBytes) + // Mid-frame, the images older than this frame are the ones the rest of it samples + // again; evicting them below the critical level only recreates and uploads them. + if (endsFrame) + { + Collect(tick, allowAggressive: false); + } + + if (CollectionMemoryBytes >= _criticalMemoryBytes) { Collect(tick, allowAggressive: true); } } + public void ReleaseUnusedMemory() + { + using var held = _lock.Hold(); + ReleaseUnusedMemoryCore(); + } + + private void ReleaseUnusedMemoryCore() + { + // Release retained image objects before trimming the allocator blocks + // containing them. Neither operation touches a registered image. + _backingPool?.ReleaseRetained(); + _imageMemoryPool.ReleaseRetained(); + _device.Slabs.ReleaseUnused(); + _tiler.ReleaseUnusedScratch(); + } + private void Collect(ulong tick, bool allowAggressive) { - var pressured = _totalUsedMemory >= _memoryPressureBytes; - var aggressive = allowAggressive && _totalUsedMemory >= _criticalMemoryBytes; - var age = aggressive ? 160UL : pressured ? 80UL : 16UL; - if (tick < age && !pressured && !aggressive) - { - return; - } - - age = Math.Min(age, tick); + var pressured = CollectionMemoryBytes >= _memoryPressureBytes; + var aggressive = allowAggressive && CollectionMemoryBytes >= _criticalMemoryBytes; + // Under pressure, retain the current collection interval's bindings only. + // Recorded GPU references are protected by deferred destruction in DeleteImage; + // waiting more frames as memory fills prevents reclamation in slow titles. + var minimumAge = pressured ? 1UL : 16UL; + if (tick < minimumAge) return; + var age = minimumAge; var deletions = aggressive ? 40 : pressured ? 20 : 10; var candidates = new List(deletions); // Deleting a depth image also deletes its stencil association, so the recency walk ends first. _recencyQueue.ForEachItemAtOrBeforeTick(tick - age, imageIdentifier => { candidates.Add(imageIdentifier); - return candidates.Count == deletions; + // A protected prefix must not hide older reclaimable images behind it. + // Stop after successful retirements below, not after visiting candidates. + return false; }); foreach (var imageIdentifier in candidates) { - if (deletions == 0) + if (deletions <= 0) { break; } - deletions--; + if (pressured && _scheduler.Active && !_scheduler.InsideTickCallback && + _tiler.ScratchOverBudget) + { + // Readback scratch and retired images are still resident until + // completion. Bound the batch before allocating the next copy. + // Completion publishes guest data and can re-enter this cache. + _lock.Exit(); + try { _scheduler.Finish(); } + finally { _lock.Enter(); } + ReleaseUnusedMemoryCore(); + } + var owner = _slots.TryGet(imageIdentifier); if (owner == null || !owner.Registered || owner.DepthOwner.IsValid) { @@ -65,8 +217,16 @@ public sealed partial class GuestImageCache if (owner.IsGpuModified) { + if (owner.IsMaybeCpuDirty) + { + if (owner.NeedsMaybeCpuHash) + continue; + _ = owner.ResolveMaybeCpuHash(owner.HashGuestEdges()); + } + var safe = CanReadBack(owner); - if (safe && owner.Description.IsTiled) + // A dirty buffer overlap does not prove that it replaces the image contents. + if (!safe && owner.SafeToDownload) { continue; } @@ -81,17 +241,18 @@ public sealed partial class GuestImageCache continue; } - owner.ClearGpuModified(); + // Only the safe path published the contents first; the other one loses them. + owner.ClearGpuModified(safe ? "EvictionAfterDownload" : "EvictionWithoutDownload", 0); } - DeleteImage(imageIdentifier); - if (_totalUsedMemory < _criticalMemoryBytes && aggressive) + deletions -= DeleteImage(imageIdentifier); + if (CollectionMemoryBytes < _criticalMemoryBytes && aggressive) { deletions >>= 2; aggressive = false; } - if (_totalUsedMemory < _memoryPressureBytes && pressured) + if (CollectionMemoryBytes < _memoryPressureBytes && pressured) { deletions >>= 1; pressured = false; @@ -118,6 +279,7 @@ public sealed partial class GuestImageCache // Test seams: thresholds, the recency order and the private state the tests inspect. internal void SetCollectionThresholds(ulong trigger, ulong pressure, ulong critical, ulong tick) { + _collectionThresholdsOverridden = true; _collectionStartBytes = trigger; _memoryPressureBytes = pressure; _criticalMemoryBytes = critical; @@ -262,7 +424,7 @@ public sealed partial class GuestImageCache } metadata.Kind = SurfaceMetadataKind.HTile; - metadata.ClearMask = 0; + metadata.SetFromMask(0); } internal void RegisterDccMetadataForTest(ulong address, ulong size) diff --git a/src/SharpEmu.Libs/Gpu/Images/GuestImageCache.Lookup.cs b/src/SharpEmu.Libs/Gpu/Images/GuestImageCache.Lookup.cs index 807c88c..646c426 100644 --- a/src/SharpEmu.Libs/Gpu/Images/GuestImageCache.Lookup.cs +++ b/src/SharpEmu.Libs/Gpu/Images/GuestImageCache.Lookup.cs @@ -8,7 +8,7 @@ namespace SharpEmu.Libs.Gpu.Images; public sealed partial class GuestImageCache { - private const int LookupSlots = 256; + private const int LookupSlots = 4096; private static readonly bool LookupMemoEnabled = Environment.GetEnvironmentVariable("SHARPEMU_IMAGE_LOOKUP_MEMO") != "0"; diff --git a/src/SharpEmu.Libs/Gpu/Images/GuestImageCache.Metadata.cs b/src/SharpEmu.Libs/Gpu/Images/GuestImageCache.Metadata.cs index f7aa599..e50a7a7 100644 --- a/src/SharpEmu.Libs/Gpu/Images/GuestImageCache.Metadata.cs +++ b/src/SharpEmu.Libs/Gpu/Images/GuestImageCache.Metadata.cs @@ -9,7 +9,7 @@ using Silk.NET.Vulkan; namespace SharpEmu.Libs.Gpu.Images; // Surface metadata (HTile, DCC, CMask, FMask) keyed by its guest address. -public sealed partial class GuestImageCache +public sealed unsafe partial class GuestImageCache { private unsafe void MaterializeDccClear( ResourceSlotIdentifier imageIdentifier, @@ -276,17 +276,24 @@ public sealed partial class GuestImageCache { fillValue = 0; using var held = _lock.Hold(); - if (!_surfaceMetadata.TryGetValue(address, out var found) || found.Kind == SurfaceMetadataKind.PendingDcc || slice >= 32) + if (!_surfaceMetadata.TryGetValue(address, out var found) || found.Kind == SurfaceMetadataKind.PendingDcc) { return false; } fillValue = found.FillValue; - return (found.ClearMask & (1u << (int)slice)) != 0; + return found.IsSliceCleared(slice); } public bool IsMetadataCleared(ulong address, uint slice) => IsMetadataCleared(address, slice, out _); + private static SurfaceMetadata CreateMetadata(SurfaceMetadataKind kind, uint clearMask, uint fillValue = 0xffffffff, ulong fillSize = 0) + { + var metadata = new SurfaceMetadata { Kind = kind, FillValue = fillValue, FillSize = fillSize }; + metadata.SetFromMask(clearMask); + return metadata; + } + public bool OverlapsDccMetadata(ulong address, ulong size) { using var held = _lock.Hold(); @@ -311,7 +318,7 @@ public sealed partial class GuestImageCache return false; } - found.ClearMask = uint.MaxValue; + found.SetFromMask(uint.MaxValue); return true; } @@ -333,13 +340,13 @@ public sealed partial class GuestImageCache if (!_surfaceMetadata.TryGetValue(address, out var found)) { // The fill may precede color-target discovery; a pending entry stays invisible until then. - _surfaceMetadata.Add(address, new SurfaceMetadata { Kind = SurfaceMetadataKind.PendingDcc, ClearMask = dccClearMask, FillValue = fillValue, FillSize = size }); + _surfaceMetadata.Add(address, CreateMetadata(SurfaceMetadataKind.PendingDcc, dccClearMask, fillValue, size)); return false; } if (found.Kind == SurfaceMetadataKind.PendingDcc) { - found.ClearMask = dccClearMask; + found.SetFromMask(dccClearMask); found.FillValue = fillValue; found.FillSize = size; return false; @@ -347,7 +354,7 @@ public sealed partial class GuestImageCache if (found.Kind == SurfaceMetadataKind.Dcc) { - found.ClearMask = dccClearMask; + found.SetFromMask(dccClearMask); found.FillValue = fillValue; found.FillSize = size; return true; @@ -356,6 +363,64 @@ public sealed partial class GuestImageCache return false; } + // A shader read of a DCC-compressed surface decompresses through the metadata its descriptor + // names, so a fast-cleared surface reads back its clear color. A surface cleared and then only + // sampled never binds as a target again: the pending clear lands here, in command order before + // the read. The register code (0x20) needs the target's clear words and stays with the target bind. + public void ApplyPendingDccClear(ResourceSlotIdentifier imageIdentifier, ulong metadataAddress) + { + using var held = _lock.Hold(); + var image = _slots[imageIdentifier]; + if (image.Description.IsVolume || !image.Backing.Exists || + !_surfaceMetadata.TryGetValue(metadataAddress, out var metadata) || + metadata.Kind != SurfaceMetadataKind.Dcc || + !ImageRequestBuilders.SupportsDccFixedClear(image.Description.PixelFormat) || + !ImageRequestBuilders.TryFixedDccClearValue((byte)metadata.FillValue, out var value)) + { + return; + } + + // Only the slices the image holds; the fill also covers slices of a larger surface. + var layers = Math.Min(image.Backing.Layers, 32u); + var sliceMask = layers == 32 ? uint.MaxValue : (1u << (int)layers) - 1; + if ((metadata.ClearMask & sliceMask) == 0) + { + return; + } + + var command = _scheduler.Current; + if (command.IsInvalid) + { + throw SubmissionScheduler.Fatal($"A pending DCC clear has no command buffer: address=0x{image.Description.Data.Address:X16}."); + } + + command.EndRendering(); + var native = new CommandBuffer(command.Handle); + for (var layer = 0u; layer < layers;) + { + if ((metadata.ClearMask & (1u << (int)layer)) == 0) + { + layer++; + continue; + } + + var first = layer; + while (layer < layers && (metadata.ClearMask & (1u << (int)layer)) != 0) + { + metadata.ClearMask &= ~(1u << (int)layer); + layer++; + } + + var range = new SubresourceRange(0, image.Backing.MipLevels, first, layer - first); + image.Transition(ImageLayout.TransferDstOptimal, AccessFlags.TransferWriteBit, range, native); + var vkRange = new ImageSubresourceRange(ImageAspectFlags.ColorBit, 0, image.Backing.MipLevels, first, layer - first); + _device.Vk.CmdClearColorImage(native, image.Backing.Handle, ImageLayout.TransferDstOptimal, &value, 1, &vkRange); + } + + WatchImage(imageIdentifier); + TakeGpuOwnership(image); + } + public static bool IsDccClearCode(byte code) => code is 0x00 or 0x20 or 0x40 or 0x80 or 0xc0; public bool TryReadGuestDccClear(ulong metadataAddress, ulong sliceSize, uint slice, out ulong sliceAddress, out byte code) @@ -423,20 +488,7 @@ public sealed partial class GuestImageCache public bool SetMetadataSlice(ulong address, uint slice, bool isClear) { using var held = _lock.Hold(); - if (!_surfaceMetadata.TryGetValue(address, out var found) || found.Kind == SurfaceMetadataKind.PendingDcc || slice >= 32) - { - return false; - } - - if (isClear) - { - found.ClearMask |= 1u << (int)slice; - } - else - { - found.ClearMask &= ~(1u << (int)slice); - } - - return true; + return _surfaceMetadata.TryGetValue(address, out var found) && found.Kind != SurfaceMetadataKind.PendingDcc && + found.SetSlice(slice, isClear); } } diff --git a/src/SharpEmu.Libs/Gpu/Images/GuestImageCache.Overlap.cs b/src/SharpEmu.Libs/Gpu/Images/GuestImageCache.Overlap.cs index f61d4b8..955c8f2 100644 --- a/src/SharpEmu.Libs/Gpu/Images/GuestImageCache.Overlap.cs +++ b/src/SharpEmu.Libs/Gpu/Images/GuestImageCache.Overlap.cs @@ -236,6 +236,47 @@ public sealed partial class GuestImageCache destination.ClearBufferModified(); } + // Two formats of one surface with the same layout are separate images over the same guest + // bytes, which the GPU writes through either of them. Before one is used, it copies the + // newest GPU contents among them, as unified guest memory would show them (the alias + // synchronization of other unified-memory emulators). Depth keeps its reinterpretation path. + private void SynchronizeAliases(ResourceSlotIdentifier imageIdentifier) + { + var image = _slots[imageIdentifier]; + if (image.Description.IsDepth || ImageDescription.IsEmptyRange(image.Description.Data)) + { + return; + } + + var newest = ResourceSlotIdentifier.Invalid; + var newestContents = image.ContentSequence; + foreach (var aliasIdentifier in FindImagesInRange(image.Description.Data.Address, image.Description.Data.Size, pageOverlap: false)) + { + if (aliasIdentifier == imageIdentifier || _slots.TryGet(aliasIdentifier) is not { } alias || !alias.Registered || + alias.DepthOwner.IsValid || !alias.SafeToDownload || alias.ContentSequence <= newestContents || + !IsSameLayoutAlias(image.Description, alias.Description)) + { + continue; + } + + newest = aliasIdentifier; + newestContents = alias.ContentSequence; + } + + if (!newest.IsValid) + { + return; + } + + CopyWholeImage(imageIdentifier, newest); + image.ContentSequence = newestContents; + } + + private static bool IsSameLayoutAlias(in ImageDescription image, in ImageDescription alias) => + !alias.IsDepth && alias.Data == image.Data && SameExtent(alias.Extent, image.Extent) && alias.Type == image.Type && + alias.Samples == image.Samples && alias.BytesPerBlock == image.BytesPerBlock && alias.TileMode == image.TileMode && + alias.Resources == image.Resources && SameMipLayout(alias, image); + private void CopyIntoMip(ResourceSlotIdentifier destinationImageIdentifier, ResourceSlotIdentifier sourceImageIdentifier, uint mip, uint layer) { RefreshCopySource(sourceImageIdentifier); @@ -421,7 +462,9 @@ public sealed partial class GuestImageCache } PrepareCopyTarget(replacement); - _blit.Reinterpret(cached, replacement); + // The conversion renders at the destination's guest extent, and a multisample + // destination never scales, so a scaled source contributes its guest-sized twin. + _blit.Reinterpret(cached.AtGuestResolution(), replacement); TakeGpuOwnership(replacement); } else @@ -433,6 +476,32 @@ public sealed partial class GuestImageCache return replacementImageIdentifier; } + private static readonly bool DropWithoutPublishing = + Environment.GetEnvironmentVariable("SHARPEMU_IMAGE_DROP_WITHOUT_PUBLISH") == "1"; + + // Memory keeps its bytes when another image or a buffer write reuses part of it. An image + // whose GPU contents never reached guest memory moves them into the buffer over its range + // before it is dropped, so a later image over those bytes starts from them, not from stale + // guest memory. False when the contents cannot be copied; they are lost as before. + private bool PublishBeforeDrop(ResourceSlotIdentifier imageIdentifier) + { + var image = _slots[imageIdentifier]; + if (DropWithoutPublishing || !image.IsGpuModified || image.DepthOwner.IsValid || !image.SafeToDownload) + { + return false; + } + + var range = image.Description.Data; + if (_bufferCache.ObtainBufferForImageWriteBack(range.Address, range.Size) is not { } buffer || + !TryDownloadImageToBuffer(image, buffer)) + { + return false; + } + + image.ClearGpuModified("PublishedToBuffer", 0); + return true; + } + private OverlapResolution ResolveOverlap(in ImageDescription requested, ImageRole role, ResourceSlotIdentifier cachedImageIdentifier, ResourceSlotIdentifier mergedImageIdentifier) { using var profileScope = RenderPhaseProfile.MeasureDetail(RenderPhaseProfile.Phase.ImageOverlap); @@ -442,6 +511,11 @@ public sealed partial class GuestImageCache return new OverlapResolution(mergedImageIdentifier); } + // Resolving can allocate a replacement, and an allocation under memory pressure collects + // images not used this tick. The image being resolved is the copy source of that + // replacement; touching it keeps it out of the collection. + TouchImage(cached); + ref var cachedInfo = ref cached.Description; var currentTick = _scheduler.CurrentTick; var safeToDelete = currentTick - Math.Min(currentTick, cached.LastAccessTick) > TicksBeforeRemoval; @@ -475,6 +549,7 @@ public sealed partial class GuestImageCache { if (safeToDelete) { + PublishBeforeDrop(cachedImageIdentifier); ReleaseImage(cachedImageIdentifier); } @@ -531,6 +606,7 @@ public sealed partial class GuestImageCache if (requested.Data.Address >= cachedInfo.Data.Address && safeToDelete) { + PublishBeforeDrop(cachedImageIdentifier); ReleaseImage(cachedImageIdentifier); } @@ -600,13 +676,41 @@ public sealed partial class GuestImageCache } var association = ResourceSlotIdentifier.Invalid; + var stale = new List(); foreach (var imageIdentifier in FindImagesInRange(stencil.Address, stencil.Size, pageOverlap: false)) { var owner = _slots.TryGet(imageIdentifier); - if (owner != null && owner.Description.Data.Address == stencil.Address) + if (owner == null || owner.Description.Data.Address != stencil.Address) + { + continue; + } + + // Only an image of exactly this span can stand for the plane: the stencil upload + // detiles the whole depth extent from it. A stencil proxy of another span is left + // over from an earlier surface at this address and is replaced; any other image of + // another size (a texture reading part of the plane) is a separate resource. + if (owner.Description.Data.Size == stencil.Size) { association = imageIdentifier; } + else if (owner.DepthOwner.IsValid) + { + stale.Add(imageIdentifier); + } + } + + // The replaced plane belongs to another depth image. What the GPU wrote to it reaches + // guest memory before the association goes, as when a depth image changes planes. + foreach (var imageIdentifier in stale) + { + var proxy = _slots[imageIdentifier]; + if (proxy.IsGpuModified) + { + WriteBackStencilPlane(_slots[proxy.DepthOwner], proxy.Description.Data); + proxy.ClearGpuModified(); + } + + DeleteImage(imageIdentifier); } if (!association.IsValid) @@ -618,6 +722,7 @@ public sealed partial class GuestImageCache } var record = _slots[association]; + TouchImage(record); record.AssociateDepth(depthImageIdentifier); return association; diff --git a/src/SharpEmu.Libs/Gpu/Images/GuestImageCache.Tracking.cs b/src/SharpEmu.Libs/Gpu/Images/GuestImageCache.Tracking.cs index 91a8bc7..d0b845a 100644 --- a/src/SharpEmu.Libs/Gpu/Images/GuestImageCache.Tracking.cs +++ b/src/SharpEmu.Libs/Gpu/Images/GuestImageCache.Tracking.cs @@ -72,15 +72,16 @@ public sealed partial class GuestImageCache } // Removes the image and its stencil associations; the slot is freed after the current tick. - private void DeleteImage(ResourceSlotIdentifier imageIdentifier) + private int DeleteImage(ResourceSlotIdentifier imageIdentifier) { using var profileScope = RenderPhaseProfile.MeasureDetail(RenderPhaseProfile.Phase.ImageDelete); var image = _slots.TryGet(imageIdentifier); if (image == null || !image.Registered) { - return; + return 0; } + var retiredImages = 1; if (!image.DepthOwner.IsValid) { var associations = new List(); @@ -99,7 +100,7 @@ public sealed partial class GuestImageCache associated.ClearGpuModified(); } - DeleteImage(association); + retiredImages += DeleteImage(association); } } @@ -108,29 +109,44 @@ public sealed partial class GuestImageCache throw SubmissionScheduler.Fatal($"A GPU-modified image cannot be deleted before its contents are resolved: address=0x{image.Description.Data.Address:X16} size=0x{image.Description.Data.Size:X}."); } + if (BindlessImageInvalidator is { } invalidate) + { + var views = image.GetOwnedViews().ToArray(); + if (views.Length != 0) invalidate(views); + } + _scheduledReadbacks.Remove(imageIdentifier); if (image.Description.HasMetadata) { _surfaceMetadata.Remove(image.Description.Metadata.Range.Address); } + var retiredBytes = checked((long)image.AccountedSize); RemoveFromIndex(imageIdentifier); if (_scheduler.Active) { - _scheduler.QueueCompletionAction(() => _slots.Erase(imageIdentifier)); + Interlocked.Add(ref _retiredImageMemoryBytes, retiredBytes); + _scheduler.QueueCompletionAction(() => + { + _slots.Erase(imageIdentifier); + Interlocked.Add(ref _retiredImageMemoryBytes, -retiredBytes); + }); } else { _slots.Erase(imageIdentifier); } + return retiredImages; } - private void ReleaseImage(ResourceSlotIdentifier imageIdentifier) + private void ReleaseImage(ResourceSlotIdentifier imageIdentifier, + [System.Runtime.CompilerServices.CallerFilePath] string file = "", + [System.Runtime.CompilerServices.CallerLineNumber] int line = 0) { var image = _slots[imageIdentifier]; if (image.IsGpuModified) { - image.ClearGpuModified(); + image.ClearGpuModified(file, line); } DeleteImage(imageIdentifier); @@ -465,20 +481,56 @@ public sealed partial class GuestImageCache return; } + using var held = _lock.Hold(); + ImageDropTrace.CauseAddress = address; + ImageDropTrace.CauseSize = size; + try + { + foreach (var imageIdentifier in FindImagesInRange(address, size, pageOverlap: true)) + { + var image = _slots[imageIdentifier]; + if (!image.Overlaps(address, size)) + { + continue; + } + + if (image.IsGpuModified) + { + // A partial write leaves the rest of the image's bytes in memory. + var range = image.Description.Data; + var fullyCovered = address <= range.Address && address + size >= range.Address + range.Size; + if (fullyCovered || !PublishBeforeDrop(imageIdentifier)) + { + image.ClearGpuModified(); + } + } + + image.MarkBufferModified(); + } + } + finally + { + ImageDropTrace.CauseSize = 0; + } + } + + // A GPU-modified image owns its bytes; a raw binding can overlap it without writing there. + public void InvalidateMemoryCopiesFromGpu(ulong address, ulong size) + { + if (!IsValidRange(address, size)) + { + return; + } + using var held = _lock.Hold(); foreach (var imageIdentifier in FindImagesInRange(address, size, pageOverlap: true)) { var image = _slots[imageIdentifier]; - if (!image.Overlaps(address, size)) + if (image.DepthOwner.IsValid || image.IsGpuModified || !image.Overlaps(address, size)) { continue; } - if (image.IsGpuModified) - { - image.ClearGpuModified(); - } - image.MarkBufferModified(); } } @@ -506,6 +558,54 @@ public sealed partial class GuestImageCache return new ImageRegionInfo(imagePages, imageBytes, gpuImageBytes); } + // The GPU-modified images whose contents are not yet in a buffer and that overlap the range. + // An image the CPU or a buffer wrote over since holds superseded contents; memory is newer. + // The ranges' test without building them: resource reads ask it for every guest word + // and almost never meet such an image. + public bool HasUnsynchronizedGpuImage(ulong address, ulong size) + { + if (!IsValidRange(address, size)) return false; + using var held = _lock.Hold(); + if (!ImagePageOwnerTable.TryGetPageRange(address, size, out var first, out var lastExclusive)) return false; + for (var page = first; page < lastExclusive; page++) + { + var owners = _pageOwners.Find(page); + if (owners is null) continue; + for (var ownerIndex = 0; ownerIndex < owners.Count; ownerIndex++) + { + if (IsUnsynchronizedGpuImage(_slots.TryGet(owners[ownerIndex]), address, size)) return true; + } + } + + return false; + } + + private static bool IsUnsynchronizedGpuImage(CachedImage? image, ulong address, ulong size) => + image is not null && !image.DepthOwner.IsValid && image.GpuOverlaps(address, size) && !image.BufferHoldsGpuContents && + !image.IsCpuDirty && !image.IsBufferModified; + + public List<(ulong Address, ulong Size)> UnsynchronizedGpuImageRanges(ulong address, ulong size) + { + var ranges = new List<(ulong Address, ulong Size)>(); + if (!IsValidRange(address, size)) return ranges; + using var held = _lock.Hold(); + if (!ImagePageOwnerTable.TryGetPageRange(address, size, out var first, out var lastExclusive)) return ranges; + for (var page = first; page < lastExclusive; page++) + { + var owners = _pageOwners.Find(page); + if (owners is null) continue; + for (var ownerIndex = 0; ownerIndex < owners.Count; ownerIndex++) + { + var image = _slots.TryGet(owners[ownerIndex]); + if (IsUnsynchronizedGpuImage(image, address, size) && + !ranges.Contains((image!.Description.Data.Address, image.Description.Data.Size))) + ranges.Add((image.Description.Data.Address, image.Description.Data.Size)); + } + } + + return ranges; + } + // Only byte overlap with a GPU-owned image can make a clean backing read unsafe. public bool HasGpuModifiedImageBytes(ulong address, ulong size) { diff --git a/src/SharpEmu.Libs/Gpu/Images/GuestImageCache.Transfers.cs b/src/SharpEmu.Libs/Gpu/Images/GuestImageCache.Transfers.cs index 000513d..c6858fa 100644 --- a/src/SharpEmu.Libs/Gpu/Images/GuestImageCache.Transfers.cs +++ b/src/SharpEmu.Libs/Gpu/Images/GuestImageCache.Transfers.cs @@ -1,6 +1,7 @@ // Copyright (C) 2026 SharpEmu Emulator Project // SPDX-License-Identifier: GPL-2.0-or-later +using SharpEmu.HLE.GpuMemory; using SharpEmu.Libs.Gpu.Buffers; using SharpEmu.Libs.Gpu.Scheduling; using SharpEmu.Libs.Gpu.Vulkan; @@ -28,6 +29,7 @@ public sealed unsafe partial class GuestImageCache public bool Tiled; public bool SwapBgra16; public bool Valid; + public TileElementLayout Element; } private sealed class ImageDownloadPlan @@ -48,6 +50,27 @@ public sealed unsafe partial class GuestImageCache return size; } + // A single-level linear target whose builder laid out one slice with the guest's pitch, + // which need not be aligned like a texture's. Textures leave the layout empty, and other + // shapes keep the format's linear layout. + private static MipLevelLayout[] GuestLinearLevels(in ImageDescription info, uint levels) + { + if (info.TileMode != GuestTileMode.Linear || levels != 1 || info.IsVolume || info.Resources.Layers == 0) + { + return []; + } + + var slice = info.MipLayout[0]; + if (slice.Size == 0 || slice.Pitch < info.Extent.Width || slice.Height < info.Extent.Height || + (ulong)slice.Pitch * slice.Height * info.BytesPerBlock > slice.Size || + slice.Size * info.Resources.Layers > info.Data.Size) + { + return []; + } + + return [slice]; + } + private static ColorTransferPlan PlanColorTransfer(CachedImage image, ImageRole role, TransferDirection direction) { ref readonly var info = ref image.Description; @@ -107,7 +130,23 @@ public sealed unsafe partial class GuestImageCache plan.SwapBgra16 = info.Bgra16; } - plan.Layout = TextureTransferLayout.Compute(format, info.Extent.Width, info.Extent.Height, info.Resources.Levels, layers, info.TileMode, info.Data.Size, allowDepthTile, volume, owner); + var guestLevels = info.FirstLevel + info.Resources.Levels; + var guestLinearLevels = GuestLinearLevels(info, guestLevels); + plan.Layout = TextureTransferLayout.Compute(format, info.Extent.Width, info.Extent.Height, guestLevels, layers, info.TileMode, info.Data.Size, allowDepthTile, volume, owner, + guestLinearLevels); + if (guestLinearLevels.Length == 0 && role == ImageRole.ColorTarget && info.TileMode == GuestTileMode.Linear && info.FirstLevel == 0 && info.Resources.Levels == 1 && !volume) + { + // Linear color targets use the target register's pitch, not texture row alignment. + var slice = checked((ulong)info.Pitch * info.Extent.Height * info.BytesPerBlock); + if (slice > info.Data.Size / layers) + throw SubmissionScheduler.Fatal("The linear color target exceeds its guest allocation."); + plan.Layout.Pitch = info.Pitch; + plan.Layout.SliceStride = info.Data.Size / layers; + plan.Layout.Mips[0] = new TransferMipLayout + { + Size = slice, RowLength = info.Pitch, ImageHeight = info.Extent.Height, + }; + } plan.Regions = plan.Layout.BuildCopies(); if (info.IsDepth) { @@ -119,9 +158,13 @@ public sealed unsafe partial class GuestImageCache } } plan.Tiled = plan.Layout.Surface.Description.TileMode != GuestTileMode.Linear; + if (!plan.Tiled && !TileGeometry.TryGetElementLayout(format, out plan.Element)) + { + return plan; + } if (plan.Tiled) { - if (!plan.Layout.TryBuildTileTransfers(info.Data.Size, plan.Regions, info.Resources.Levels, out var tiles)) + if (!plan.Layout.TryBuildTileTransfers(info.Data.Size, plan.Regions, guestLevels, out var tiles)) { return plan; } @@ -130,6 +173,19 @@ public sealed unsafe partial class GuestImageCache plan.LinearSize = LinearSizeOf(plan.Tiles); } + // Only the resident mips have host levels: guest level L lands on host level L - FirstLevel. + var firstLevel = info.FirstLevel; + if (firstLevel != 0) + { + plan.Regions.RemoveAll(region => region.ImageSubresource.MipLevel < firstLevel); + for (var index = 0; index < plan.Regions.Count; index++) + { + var region = plan.Regions[index]; + region.ImageSubresource.MipLevel -= firstLevel; + plan.Regions[index] = region; + } + } + plan.Valid = true; return plan; } @@ -138,11 +194,14 @@ public sealed unsafe partial class GuestImageCache SubmissionScheduler.Fatal( $"The color-attachment upload is invalid: address=0x{info.Data.Address:X16} size=0x{info.Data.Size:X} layers={info.Resources.Layers} samples={info.Samples} backingSamples={image.Backing.Samples} compression={info.Metadata.Compression}."); + private static int _linearReadbackOverflowWarned; + private static ImageDownloadPlan PlanDownload(CachedImage image) { ref readonly var info = ref image.Description; var plan = new ImageDownloadPlan { Depth = info.IsDepth }; - if (info.Samples != 1 || image.Backing.Samples != 1) + // A streamed texture holds only its resident mips; the guest chain is not all there. + if (info.Samples != 1 || image.Backing.Samples != 1 || info.FirstLevel != 0) { return plan; } @@ -161,6 +220,20 @@ public sealed unsafe partial class GuestImageCache plan.Color = PlanColorTransfer(image, UploadRole(image), TransferDirection.Download); plan.Valid = plan.Color.Valid; + // A linear layout computed for the format can need more bytes than the guest + // range holds. Copying it would write past the readback's storage. + if (plan.Valid && !plan.Color.Tiled && + TextureTransferLayout.CopyFootprint(plan.Color.Regions, plan.Color.Element) > info.Data.Size) + { + plan.Valid = false; + if (Interlocked.Exchange(ref _linearReadbackOverflowWarned, 1) == 0) + { + Console.Error.WriteLine( + $"[GPU][WARN] A linear image readback was skipped: its copy layout needs 0x{TextureTransferLayout.CopyFootprint(plan.Color.Regions, plan.Color.Element):X} bytes " + + $"but the guest range is 0x{info.Data.Size:X}. address=0x{info.Data.Address:X16} format={info.GuestFormat} extent={info.Extent.Width}x{info.Extent.Height} levels={info.Resources.Levels}."); + } + } + return plan; } @@ -230,11 +303,11 @@ public sealed unsafe partial class GuestImageCache image.UploadFromBuffer(System.Runtime.InteropServices.CollectionsMarshal.AsSpan(copies), linear.Buffer, linear.Offset, linear.Size); } - private void UploadStencilPlane(CachedImage association, GpuBuffer source, ulong sourceOffset) + // The layout of a depth image's guest stencil plane at `plane`: one byte per texel. + private static ImageDescription StencilPlaneLayout(CachedImage depth, GuestSpan plane, string operation) { - var destination = _slots[association.DepthOwner]; - var info = destination.Description; - info.Data = association.Description.Data; + var info = depth.Description; + info.Data = plane; info.GuestFormat = GuestPixelFormat.Bits8UInt; info.BytesPerBlock = 1; if (info.IsTiled) @@ -242,12 +315,89 @@ public sealed unsafe partial class GuestImageCache info.Pitch = TileGeometry.DepthPitch(info.Extent.Width, 1); } - if (info.Samples != 1 || destination.Backing.Samples != 1 || info.Resources.Layers == 0 || info.Data.Size % info.Resources.Layers != 0) + if (info.Samples != 1 || depth.Backing.Samples != 1 || info.Resources.Layers == 0 || info.Data.Size % info.Resources.Layers != 0) { throw SubmissionScheduler.Fatal( - $"The stencil upload is invalid: address=0x{info.Data.Address:X16} size=0x{info.Data.Size:X} layers={info.Resources.Layers} samples={info.Samples} backingSamples={destination.Backing.Samples}."); + $"The stencil {operation} is invalid: address=0x{info.Data.Address:X16} size=0x{info.Data.Size:X} layers={info.Resources.Layers} samples={info.Samples} backingSamples={depth.Backing.Samples}."); } + return info; + } + + // Writes the stencil aspect back to its guest plane once the current tick completes. + private void WriteBackStencilPlane(CachedImage depth, GuestSpan plane) + { + var info = StencilPlaneLayout(depth, plane, "write-back"); + var download = _bufferCache.GetUtilityBuffer(GpuBufferUsage.Download); + if (!download.TryMap(plane.Size, out var offset, 4)) + { + throw SubmissionScheduler.Fatal($"The reusable download ring cannot map the stencil plane: address=0x{plane.Address:X16} size=0x{plane.Size:X}."); + } + + download.Commit(); + var fullSliceSize = info.Data.Size / info.Resources.Layers; + var copies = DepthCopies(info, fullSliceSize, ImageAspectFlags.StencilBit); + if (info.IsTiled) + { + // Tiling fills only the plane's texels: the padding keeps its guest bytes. + if (!_backing.TryReadBacking(plane.Address, download.Mapped.Slice((int)offset, (int)plane.Size))) + { + throw SubmissionScheduler.Fatal($"The stencil plane cannot be read from guest memory: address=0x{plane.Address:X16} size=0x{plane.Size:X}."); + } + + download.Flush(offset, plane.Size); + var tiles = DepthTileTransfers(info, DepthBlock(info), fullSliceSize); + _tiler.TileImage(depth, copies, download.Handle, offset, plane.Size, plane.Size, System.Runtime.InteropServices.CollectionsMarshal.AsSpan(tiles)); + } + else + { + for (var index = 0; index < copies.Length; index++) + { + copies[index].BufferOffset += offset; + } + + depth.DownloadToBuffer(copies, download.Handle, offset, plane.Size); + } + + var barrier = new BufferMemoryBarrier2 + { + SType = StructureType.BufferMemoryBarrier2, + SrcAccessMask = AccessFlags2.MemoryWriteBit | AccessFlags2.TransferWriteBit | AccessFlags2.ShaderWriteBit, + DstAccessMask = AccessFlags2.HostReadBit, + SrcQueueFamilyIndex = Vk.QueueFamilyIgnored, + DstQueueFamilyIndex = Vk.QueueFamilyIgnored, + Buffer = download.Handle, + Offset = offset, + Size = plane.Size, + }; + _scheduler.EndRendering(); + VulkanSynchronization.PipelineBarrier(_device.Vk, new CommandBuffer(_scheduler.Current.Handle), PipelineStageFlags.AllCommandsBit, PipelineStageFlags.HostBit, 0, 0, null, 1, &barrier, 0, null); + var backing = _backing; + // Until the tick completes guest memory holds the plane this replaces; an upload of the + // range waits for it (AwaitPublishingDownloads). + var publishing = new PublishingDownload(plane.Address, plane.Size, _scheduler.CurrentTick); + lock (_publishingDownloads) _publishingDownloads.Add(publishing); + _scheduler.QueuePriorityCompletionAction(() => + { + try + { + download.Invalidate(offset, plane.Size); + if (!backing.TryWriteBacking(plane.Address, download.Mapped.Slice((int)offset, (int)plane.Size))) + { + throw SubmissionScheduler.Fatal($"The stencil plane could not be written to guest memory: address=0x{plane.Address:X16} size=0x{plane.Size:X}."); + } + } + finally + { + lock (_publishingDownloads) _publishingDownloads.Remove(publishing); + } + }); + } + + private void UploadStencilPlane(CachedImage association, GpuBuffer source, ulong sourceOffset) + { + var destination = _slots[association.DepthOwner]; + var info = StencilPlaneLayout(destination, association.Description.Data, "upload"); var fullSliceSize = info.Data.Size / info.Resources.Layers; var copies = DepthCopies(info, fullSliceSize, ImageAspectFlags.StencilBit); var linear = new TilerBufferSpan(source.Handle, sourceOffset, source.Size - sourceOffset); @@ -260,6 +410,60 @@ public sealed unsafe partial class GuestImageCache UploadRegions(destination, copies.ToList(), linear); } + private static uint ExpandedColorChannels(CachedImage image) => + image.Backing.Format is Format.R8G8B8A8Srgb or Format.R8G8B8A8Unorm + ? image.Description.GuestFormat switch + { + GuestPixelFormat.Bits8Srgb => 1u, + GuestPixelFormat.Bits8_8Srgb => 2u, + _ => 0u, + } + : 0u; + + private static List ExpandedColorRegions(List guestRegions, out ulong size) + { + var result = new List(guestRegions.Count); + size = 0; + foreach (var guest in guestRegions) + { + var wide = guest; + wide.BufferOffset = size; + var pitch = guest.BufferRowLength == 0 ? guest.ImageExtent.Width : guest.BufferRowLength; + var height = guest.BufferImageHeight == 0 ? guest.ImageExtent.Height : guest.BufferImageHeight; + size = checked(size + (ulong)pitch * height * 4); + result.Add(wide); + } + return result; + } + + private void ConvertExpandedColor(TilerBufferSpan guest, TilerBufferSpan wide, + List guestRegions, List wideRegions, uint channels, bool widen) + { + for (var index = 0; index < guestRegions.Count; index++) + { + var region = guestRegions[index]; + if (region.ImageExtent.Depth != 1 || region.ImageSubresource.LayerCount != 1) + throw SubmissionScheduler.Fatal("Expanded color transfers require one slice per copy region."); + var pitch = region.BufferRowLength == 0 ? region.ImageExtent.Width : region.BufferRowLength; + var height = region.BufferImageHeight == 0 ? region.ImageExtent.Height : region.BufferImageHeight; + var narrowRow = checked((ulong)pitch * channels); + var wideRow = checked((ulong)pitch * 4); + if (region.BufferOffset > guest.Size || wideRegions[index].BufferOffset > wide.Size) + throw SubmissionScheduler.Fatal("Expanded color transfer offset exceeds its buffer span."); + var narrow = new TilerBufferSpan(guest.Buffer, checked(guest.Offset + region.BufferOffset), guest.Size - region.BufferOffset); + var expanded = new TilerBufferSpan(wide.Buffer, checked(wide.Offset + wideRegions[index].BufferOffset), wide.Size - wideRegions[index].BufferOffset); + _tiler.ConvertNarrowColor(widen ? narrow : expanded, widen ? expanded : narrow, channels, widen, + new DepthConversionLayout + { + Width = region.ImageExtent.Width, Height = region.ImageExtent.Height, Layers = 1, + SourceRowStride = widen ? narrowRow : wideRow, + TargetRowStride = widen ? wideRow : narrowRow, + SourceSliceStride = checked((widen ? narrowRow : wideRow) * height), + TargetSliceStride = checked((widen ? wideRow : narrowRow) * height), + }); + } + } + private void UploadFromBuffer(CachedImage image, in ImageRequest request, GpuBuffer source, ulong sourceOffset) { if (image.DepthOwner.IsValid) @@ -282,15 +486,31 @@ public sealed unsafe partial class GuestImageCache var linear = new TilerBufferSpan(source.Handle, sourceOffset, info.Data.Size); if (plan.Tiled) { + // Detile scratch is a real device-local allocation. Evict cold + // image cache entries before requesting a large dedicated + // buffer, otherwise the cache thresholds cannot protect this + // path from a single oversized upload. + CollectForAllocation(plan.LinearSize); linear = _tiler.Detile(source.Handle, sourceOffset, info.Data.Size, plan.LinearSize, System.Runtime.InteropServices.CollectionsMarshal.AsSpan(plan.Tiles)); } if (plan.SwapBgra16) { + CollectForAllocation(linear.Size); linear = _tiler.SwapBgra16(linear); } - UploadRegions(image, plan.Regions, linear); + if (ExpandedColorChannels(image) is var channels && channels != 0) + { + var wideRegions = ExpandedColorRegions(plan.Regions, out var wideSize); + var wide = _tiler.GetScratchBuffer(wideSize); + ConvertExpandedColor(linear, wide, plan.Regions, wideRegions, channels, widen: true); + UploadRegions(image, wideRegions, wide); + } + else + { + UploadRegions(image, plan.Regions, linear); + } return; } @@ -309,6 +529,7 @@ public sealed unsafe partial class GuestImageCache if (info.TileMode != GuestTileMode.Linear) { var tiles = DepthTileTransfers(info, block, fullSliceSize); + CollectForAllocation(info.Data.Size); depthLinear = _tiler.Detile(source.Handle, sourceOffset, info.Data.Size, info.Data.Size, System.Runtime.InteropServices.CollectionsMarshal.AsSpan(tiles)); } @@ -327,7 +548,9 @@ public sealed unsafe partial class GuestImageCache throw SubmissionScheduler.Fatal($"The depth upload conversion size overflows: transferSlice={transferSlice} layers={layers}."); } - var widened = _tiler.GetScratchBuffer(transferSlice * layers); + var widenedSize = transferSlice * layers; + CollectForAllocation(widenedSize); + var widened = _tiler.GetScratchBuffer(widenedSize); _tiler.ConvertDepth16(depthLinear, widened, DepthConversionDirection.Widen, info.PixelFormat == Format.D32SfloatS8Uint, new DepthConversionLayout { Width = info.Extent.Width, @@ -358,6 +581,13 @@ public sealed unsafe partial class GuestImageCache return; } + AwaitPublishingDownloads(image.Description.Data); + if (!ReferenceEquals(_slots.TryGet(imageIdentifier), image)) + { + // The image went away while the cache waited. + return; + } + var measureUpload = RenderPhaseProfile.ImageUploadDetailsEnabled; var watchStarted = measureUpload ? System.Diagnostics.Stopwatch.GetTimestamp() : 0; WatchImage(imageIdentifier); @@ -557,9 +787,31 @@ public sealed unsafe partial class GuestImageCache } var color = plan.Color; + if (ExpandedColorChannels(image) is var channels && channels != 0) + { + var wideRegions = ExpandedColorRegions(color.Regions, out var wideSize); + var wide = _tiler.GetScratchBuffer(wideSize); + image.DownloadToBuffer(System.Runtime.InteropServices.CollectionsMarshal.AsSpan(wideRegions), wide.Buffer, 0, wide.Size); + var guest = color.Tiled + ? _tiler.Detile(destination.Handle, destinationOffset, destinationSize, color.LinearSize, + System.Runtime.InteropServices.CollectionsMarshal.AsSpan(color.Tiles)) + : new TilerBufferSpan(destination.Handle, destinationOffset, destinationSize); + ConvertExpandedColor(guest, wide, color.Regions, wideRegions, channels, widen: false); + if (color.Tiled) + _tiler.Tile(guest.Buffer, guest.Offset, guest.Size, destination.Handle, destinationOffset, destinationSize, + System.Runtime.InteropServices.CollectionsMarshal.AsSpan(color.Tiles)); + return; + } var swap = color.SwapBgra16 ? ColorChannelSwap.SwapBgra16 : ColorChannelSwap.None; if (!color.Tiled) { + var footprint = TextureTransferLayout.CopyFootprint(color.Regions, color.Element); + if (footprint > destinationSize) + { + throw SubmissionScheduler.Fatal( + $"A linear image download exceeds its destination: needed=0x{footprint:X} destination=0x{destinationSize:X} address=0x{image.Description.Data.Address:X16}."); + } + if (swap == ColorChannelSwap.SwapBgra16) { var linear = _tiler.GetScratchBuffer(destinationSize); @@ -624,17 +876,51 @@ public sealed unsafe partial class GuestImageCache } } - if (!selected.IsValid) + var synchronized = false; + if (selected.IsValid) { - return false; + var target = selected; + if (_slots.TryGet(target) is { DepthOwner.IsValid: true } proxy) + { + target = proxy.DepthOwner; + } + + synchronized = TryDownloadImageToBuffer(_slots[target], buffer); } - if (_slots.TryGet(selected) is { DepthOwner.IsValid: true } proxy) + // Every other image the view covers is memory the shader can read or partly + // rewrite: its GPU-only contents move into the buffer before the bind hands + // the range to the buffer. Aliased images are left out, their order unknown. + var covered = new List(); + foreach (var identifier in FindImagesInRange(address, size, pageOverlap: false)) { - selected = proxy.DepthOwner; + if (identifier != selected && _slots.TryGet(identifier) is { DepthOwner.IsValid: false } candidate && + candidate.Description.Data.Address >= address && + candidate.Description.Data.Size <= address + size - candidate.Description.Data.Address) + { + covered.Add(identifier); + } } - var image = _slots[selected]; + foreach (var identifier in covered) + { + var image = _slots[identifier]; + if (image.BufferHoldsGpuContents || !CanReadBack(image) || + (selected.IsValid && _slots[selected].Overlaps(image.Description.Data.Address, image.Description.Data.Size)) || + covered.Any(other => other != identifier && + _slots[other].Overlaps(image.Description.Data.Address, image.Description.Data.Size))) + { + continue; + } + + synchronized |= TryDownloadImageToBuffer(image, buffer); + } + + return synchronized; + } + + private bool TryDownloadImageToBuffer(CachedImage image, GpuBuffer buffer) + { ref readonly var info = ref image.Description; if (!buffer.IsInBounds(info.Data.Address, 1)) { @@ -703,11 +989,46 @@ public sealed unsafe partial class GuestImageCache } } + // The kept levels must fit the bytes the buffer covers, as in a full readback. + if (!plan.Depth && !plan.Color.Tiled && + TextureTransferLayout.CopyFootprint(plan.Color.Regions, plan.Color.Element) > copySize) + { + return false; + } + + // The mip layout can end before the tile plan does (Silent Hill: a 128x128 slice planned at + // the exact end of the mip chain aborted the process). Cover the whole plan when the buffer + // holds it; otherwise keep the guest bytes instead of copying a partial image. + if (!plan.Depth && plan.Color.Tiled) + { + var tiledEnd = TiledEndOf(plan.Color.Tiles); + if (tiledEnd > copySize) + { + if (tiledEnd > available) + { + return false; + } + + copySize = tiledEnd; + } + } + DownloadToBuffer(image, buffer, bufferOffset, copySize, plan); image.MarkBufferHoldsGpuContents(); return true; } + private static ulong TiledEndOf(List tiles) + { + var end = 0UL; + foreach (var tile in tiles) + { + end = Math.Max(end, tile.TiledOffset + tile.TiledSize); + } + + return end; + } + // Publishes a GPU-owned image to guest memory after its tick completes; false when it cannot. private bool TryDownloadToGuest(ResourceSlotIdentifier imageIdentifier) { @@ -758,12 +1079,21 @@ public sealed unsafe partial class GuestImageCache _scheduler.EndRendering(); VulkanSynchronization.PipelineBarrier(_device.Vk,new CommandBuffer(_scheduler.Current.Handle), PipelineStageFlags.AllCommandsBit, PipelineStageFlags.HostBit, 0, 0, null, 1, &barrier, 0, null); var backing = _backing; + var publishing = new PublishingDownload(range.Address, range.Size, _scheduler.CurrentTick); + lock (_publishingDownloads) _publishingDownloads.Add(publishing); _scheduler.QueuePriorityCompletionAction(() => { - download.Invalidate(offset, range.Size); - if (!backing.TryWriteBacking(range.Address, download.Mapped.Slice((int)offset, (int)range.Size))) + try { - throw SubmissionScheduler.Fatal($"The image readback could not be written to guest memory: address=0x{range.Address:X16} size=0x{range.Size:X}."); + download.Invalidate(offset, range.Size); + if (!backing.TryWriteBacking(range.Address, download.Mapped.Slice((int)offset, (int)range.Size))) + { + throw SubmissionScheduler.Fatal($"The image readback could not be written to guest memory: address=0x{range.Address:X16} size=0x{range.Size:X}."); + } + } + finally + { + lock (_publishingDownloads) _publishingDownloads.Remove(publishing); } }); // Completion actions wait for the priority readback before freeing spill buffers. @@ -771,6 +1101,45 @@ public sealed unsafe partial class GuestImageCache return true; } + private sealed record PublishingDownload(ulong Address, ulong Size, ulong Tick); + + // Readbacks whose guest bytes are written only when their tick completes. Until then guest + // memory holds the contents they supersede. + private readonly List _publishingDownloads = new(); + + // Waits until every readback that will publish into the range has written guest memory, so + // an upload does not read the bytes it replaces. The cache lock is released while waiting. + private void AwaitPublishingDownloads(GuestSpan range) + { + ulong tick = 0; + lock (_publishingDownloads) + { + foreach (var download in _publishingDownloads) + { + if (download.Address < range.Address + range.Size && range.Address < download.Address + download.Size) + tick = Math.Max(tick, download.Tick); + } + } + + if (tick == 0) + { + return; + } + + if (_scheduler.InsideTickCallback) + { + throw SubmissionScheduler.Fatal($"An image upload cannot wait for a pending readback from a completion callback: address=0x{range.Address:X16} size=0x{range.Size:X}."); + } + + _lock.Exit(); + try + { + _scheduler.Wait(tick); + _scheduler.WaitForPriorityOperations(tick); + } + finally { _lock.Enter(); } + } + // Clears the one image that owns exactly this range; false when no single owner or clear value fits. public bool TryClearImageFromBuffer(ulong address, ulong size, uint packedClear) { diff --git a/src/SharpEmu.Libs/Gpu/Images/GuestImageCache.cs b/src/SharpEmu.Libs/Gpu/Images/GuestImageCache.cs index 5568e0b..20ea334 100644 --- a/src/SharpEmu.Libs/Gpu/Images/GuestImageCache.cs +++ b/src/SharpEmu.Libs/Gpu/Images/GuestImageCache.cs @@ -17,6 +17,9 @@ public sealed unsafe partial class GuestImageCache : IGuestImageCache, IGuestIma { private const ulong TicksBeforeRemoval = 32; private const ulong MiB = 1024 * 1024; + private const ulong GiB = 1024 * MiB; + private const ulong MinimumMemorySafetyMargin = 512 * MiB; + private const ulong NoBudgetImageCacheLimit = 4 * GiB; private readonly GpuDeviceInfo _device; private readonly SubmissionScheduler _scheduler; @@ -28,20 +31,30 @@ public sealed unsafe partial class GuestImageCache : IGuestImageCache, IGuestIma private readonly IGuestBackedSpace _backing; private readonly SlotTable _slots = new(); private readonly ImageBackingPool? _backingPool; + private readonly OptimalImageMemoryPool _imageMemoryPool; private readonly ImagePageOwnerTable _pageOwners = new(); - private readonly Dictionary _nullImages = new(); + private readonly Dictionary _nullImages = new(); private RecencyQueue _recencyQueue = new(); private readonly HashSet _scheduledReadbacks = new(); private readonly SortedDictionary _surfaceMetadata = new(); private ulong _totalUsedMemory; + private long _retiredImageMemoryBytes; private ulong _collectionStartBytes; - private ulong _memoryPressureBytes = 1536 * MiB; - private ulong _criticalMemoryBytes = 3072 * MiB; + private ulong _memoryPressureBytes; + private ulong _criticalMemoryBytes; private ulong _collectionTick; + private bool _allocationCollectionBlocked; + private bool _collectionThresholdsOverridden; + private bool _collectionBudgetLogged; private uint _queryEpoch; private bool _readbackLinearImages; private bool _disposed; + // The persistent Vulkan image heap must invalidate views before this cache + // destroys an evicted image. The callback is installed only when the device + // supports null descriptors and bindless images are active. + public Action>? BindlessImageInvalidator { get; set; } + public GuestImageCache(GpuDeviceInfo device, SubmissionScheduler scheduler, PageGuard pages, GuestBufferCache bufferCache, IGuestBackedSpace backing, bool readbackLinearImages) { _device = device; @@ -50,12 +63,78 @@ public sealed unsafe partial class GuestImageCache : IGuestImageCache, IGuestIma _bufferCache = bufferCache; _backing = backing; _readbackLinearImages = readbackLinearImages; + RefreshCollectionBudget(); _blit = new ColorToMultisampleDepthBlit(device, scheduler); _tiler = new GpuTiler(device, scheduler, bufferCache.GetUtilityBuffer(GpuBufferUsage.Stream)); _backingPool = ImageBackingPool.Enabled ? new ImageBackingPool(device) : null; + _imageMemoryPool = new OptimalImageMemoryPool(device); } - public ulong TotalUsedMemory => _totalUsedMemory; + // Set once, from the VRAM free when the device starts: recomputed later, the budget would + // count the cache's own images as used memory and shrink as the cache fills. + private void RefreshCollectionBudget() + { + if (_collectionThresholdsOverridden) + { + return; + } + + var available = Math.Max(_device.DeviceLocalAvailableBytes, 1UL); + var imageBudget = ComputeImageCacheBudget(available, _device.HasMemoryBudget); + if (ImageCacheBudgetCap is { } cap) + { + imageBudget = Math.Min(imageBudget, cap); + } + + _collectionStartBytes = Math.Max(imageBudget / 2, MiB); + _memoryPressureBytes = Math.Max(imageBudget * 3 / 5, MiB); + _criticalMemoryBytes = Math.Max(imageBudget * 7 / 10, MiB); + if (!_collectionBudgetLogged) + { + Console.Error.WriteLine( + $"[LOADER][INFO] Image cache budget source={(_device.HasMemoryBudget ? "VK_EXT_memory_budget" : "conservative heap fallback")} " + + $"heap={_device.DeviceLocalHeapBytes} budget={_device.DeviceLocalBudgetBytes} " + + $"usage={_device.DeviceLocalUsageBytes} available={_device.DeviceLocalAvailableBytes} " + + $"image_budget={imageBudget} thresholds={_collectionStartBytes}/{_memoryPressureBytes}/{_criticalMemoryBytes}"); + _collectionBudgetLogged = true; + } + } + + // SHARPEMU_IMAGE_CACHE_BUDGET_MB caps the image cache below its VRAM-derived budget, which + // otherwise keeps most free VRAM for cached images. A lower cap leaves room for other + // VRAM users, such as a frame-capture tool's copies of every resource. + private static readonly ulong? ImageCacheBudgetCap = + ulong.TryParse(Environment.GetEnvironmentVariable("SHARPEMU_IMAGE_CACHE_BUDGET_MB"), out var megabytes) && megabytes > 0 + ? megabytes * MiB + : null; + + internal static ulong ComputeImageCacheBudget(ulong available, bool hasMemoryBudget) + { + available = Math.Max(available, 1UL); + var safetyMargin = Math.Max(available / 4, MinimumMemorySafetyMargin); + var imageBudget = available > safetyMargin + ? available - safetyMargin + : Math.Max(available / 2, MiB); + if (!hasMemoryBudget) + { + // Without VK_EXT_memory_budget the physical heap size says nothing + // about allocations already held by the driver or other processes. + // Keep the historical conservative cap instead of treating all VRAM + // as available to the image cache. + imageBudget = Math.Min(imageBudget, NoBudgetImageCacheLimit); + } + + return imageBudget; + } + + public ulong TotalUsedMemory => _totalUsedMemory + (ulong)Interlocked.Read(ref _retiredImageMemoryBytes); + + public bool MemoryUnderPressure => CollectionMemoryBytes >= _memoryPressureBytes; + + public bool RetirementOverBudget => Interlocked.Read(ref _retiredImageMemoryBytes) > 256L * 1024 * 1024 || + (Interlocked.Read(ref _retiredImageMemoryBytes) > 0 && CollectionMemoryBytes >= _criticalMemoryBytes); + + public bool ScratchOverBudget => _tiler.ScratchOverBudget; public int ImageCount => _slots.Count; @@ -98,6 +177,7 @@ public sealed unsafe partial class GuestImageCache : IGuestImageCache, IGuestIma _disposed = true; _slots.ForEach((_, image) => image.Dispose()); _backingPool?.Dispose(); + _imageMemoryPool.ReleaseRetained(); _tiler.Dispose(); _blit.Dispose(); } @@ -116,6 +196,15 @@ public sealed unsafe partial class GuestImageCache : IGuestImageCache, IGuestIma throw SubmissionScheduler.Fatal("An image lookup needs a command buffer that is recording."); } + // Only the mips a streamed texture has mapped are resident. When the whole chain is + // mapped, the texture is the same image as any other view of that memory, such as the + // render target whose lower mip a downsample pass reads. + if (request.Description.FirstLevel != 0 && + _bufferCache.IsGuestRangeMapped(request.Description.Data.Address, request.Description.Data.Size)) + { + request = ImageRequestBuilders.WithWholeMipChain(request); + } + ValidateRequest(request); if (ImageDescription.IsEmptyRange(request.Description.Data)) { @@ -240,6 +329,14 @@ public sealed unsafe partial class GuestImageCache : IGuestImageCache, IGuestIma } } + // A compute shader indexes a storage image in guest pixels and its dispatch grid does + // not shrink with the host resolution, so a scaled image would be written with holes + // (upscale) or redundantly (downscale). Such memory drops back to guest resolution. + if (result.IsValid && request.ShaderWrite && _slots[result].IsScaled) + { + result = ReplaceScaled(result); + } + if (!result.IsValid) { result = InsertImage(request.Description); @@ -296,6 +393,18 @@ public sealed unsafe partial class GuestImageCache : IGuestImageCache, IGuestIma return image; } + public CachedImage AcquireStencilStorageImage(ResourceSlotIdentifier imageIdentifier, uint width, uint height) + { + using var held = _lock.Hold(); + var attachment = _slots[imageIdentifier]; + TouchImage(attachment); + CollectForAllocation((ulong)width * height * attachment.Backing.Layers); + var before = attachment.AccountedSize; + var storage = attachment.GetOrCreateStencilStorageImage(width, height); + _totalUsedMemory += attachment.AccountedSize - before; + return storage; + } + public ImageView AcquireTextureView(ResourceSlotIdentifier imageIdentifier, in ImageRequest request) { using var profileScope = RenderPhaseProfile.MeasureDetail(RenderPhaseProfile.Phase.ImageAcquire); @@ -308,6 +417,11 @@ public sealed unsafe partial class GuestImageCache : IGuestImageCache, IGuestIma throw SubmissionScheduler.Fatal($"A texture must be found again before its view is acquired: address=0x{image.Description.Data.Address:X16} registered={image.Registered} proxy={image.DepthOwner.IsValid} rebind={image.Binding.NeedsRebind}."); } + if (hasData) + { + SynchronizeAliases(imageIdentifier); + } + if (request.Role == ImageRole.StorageImage) { image.MarkGpuModified(); @@ -359,6 +473,7 @@ public sealed unsafe partial class GuestImageCache : IGuestImageCache, IGuestIma } TouchImage(image); + SynchronizeAliases(imageIdentifier); image.MarkGpuModified(); image.Uses.RenderTarget = true; RefreshFromGuest(imageIdentifier, request); @@ -404,6 +519,7 @@ public sealed unsafe partial class GuestImageCache : IGuestImageCache, IGuestIma } TouchImage(image); + SynchronizeAliases(imageIdentifier); image.MarkGpuModified(); image.Uses.DepthTarget = true; RefreshFromGuest(imageIdentifier, request); @@ -413,13 +529,13 @@ public sealed unsafe partial class GuestImageCache : IGuestImageCache, IGuestIma var address = request.Description.Metadata.Range.Address; if (!_surfaceMetadata.TryGetValue(address, out var metadata)) { - _surfaceMetadata.Add(address, new SurfaceMetadata { Kind = SurfaceMetadataKind.HTile, ClearMask = image.Description.HtileClearMask }); + _surfaceMetadata.Add(address, CreateMetadata(SurfaceMetadataKind.HTile, image.Description.HtileClearMask)); } else if (metadata.Kind == SurfaceMetadataKind.PendingDcc) { // A pending DCC fill uses the DCC encoding; it must not become HTile state. metadata.Kind = SurfaceMetadataKind.HTile; - metadata.ClearMask = image.Description.HtileClearMask; + metadata.SetFromMask(image.Description.HtileClearMask); metadata.FillValue = 0xffffffff; metadata.FillSize = 0; } @@ -431,6 +547,11 @@ public sealed unsafe partial class GuestImageCache : IGuestImageCache, IGuestIma if (request.Description.HasStencil) { + if (image.Description.HasStencil && image.Description.Stencil != request.Description.Stencil) + { + ReleaseStencilPlane(imageIdentifier, image); + } + image.Description.Stencil = request.Description.Stencil; RefreshStencilPlane(imageIdentifier, image, request.Description.Metadata.StencilCompressed); } @@ -439,6 +560,34 @@ public sealed unsafe partial class GuestImageCache : IGuestImageCache, IGuestIma return image.GetOrCreateView(request.View); } + // The depth image holds one guest stencil plane at a time. Before it takes another one (a + // title that alternates stencil buffers under one depth buffer), the plane it holds goes + // back to guest memory when the GPU changed it, and its association is dropped, so a view + // of that plane reads guest memory instead of the depth image's new plane. + private void ReleaseStencilPlane(ResourceSlotIdentifier depthIdentifier, CachedImage depth) + { + var plane = depth.Description.Stencil; + var associations = new List(); + foreach (var imageIdentifier in FindImagesInRange(plane.Address, plane.Size, pageOverlap: false)) + { + if (_slots.TryGet(imageIdentifier) is { } candidate && candidate.DepthOwner == depthIdentifier) + { + associations.Add(imageIdentifier); + } + } + + if (associations.Any(association => _slots[association].IsGpuModified)) + { + WriteBackStencilPlane(depth, plane); + } + + foreach (var association in associations) + { + _slots[association].ClearGpuModified(); + DeleteImage(association); + } + } + // GFX10 packs the mips smaller than half a swizzle block into one block at the start of the chain. // A title can write that block through a single-level view the size of the block (Unity's // screen-space reflection blur writes mips 3 and up that way), which the cache keeps as its own @@ -582,10 +731,17 @@ public sealed unsafe partial class GuestImageCache : IGuestImageCache, IGuestIma } } + private readonly record struct NullImageKey(Format Format, GuestPixelFormat GuestFormat, GuestImageType Type, uint Samples, uint Layers); + private ResourceSlotIdentifier GetNullImage(in ImageRequest request) { - var format = request.Description.PixelFormat; - if (_nullImages.TryGetValue(format, out var found)) + var key = new NullImageKey( + request.Description.PixelFormat, + request.Description.GuestFormat, + request.Description.Type, + request.Description.Samples, + request.Description.Resources.Layers); + if (_nullImages.TryGetValue(key, out var found)) { return found; } @@ -593,23 +749,96 @@ public sealed unsafe partial class GuestImageCache : IGuestImageCache, IGuestIma var description = ImageDescription.Create(); description.PixelFormat = request.Description.PixelFormat; description.GuestFormat = request.Description.GuestFormat; - description.Type = GuestImageType.Color2D; + description.Type = request.Description.Type; description.Extent = new Extent3D(1, 1, 1); - description.Resources = SubresourceCount.Single; + description.Resources = new SubresourceCount(1, key.Layers); description.Pitch = 1; description.BytesPerBlock = Math.Max(request.Description.BytesPerBlock, 1); - description.Samples = 1; + description.Samples = Math.Max(request.Description.Samples, 1); description.TileMode = GuestTileMode.Linear; description.MipLayout[0] = new MipLevelLayout { Offset = 0, Size = description.BytesPerBlock, Pitch = 1, Height = 1 }; var imageIdentifier = InsertImage(description); - _nullImages.Add(format, imageIdentifier); + _nullImages.Add(key, imageIdentifier); return imageIdentifier; } + // Guest ranges a compute shader has written through a storage image; see the storage + // branch of FindImage. Every image over such a range keeps its guest resolution. + private readonly List _unscaledRanges = new(); + + private bool IsUnscaledRange(in GuestSpan data) + { + if (ImageDescription.IsEmptyRange(data)) + { + return false; + } + + foreach (var range in _unscaledRanges) + { + if (GuestRangeOverlap.Bytes(range, data)) + { + return true; + } + } + + return false; + } + + // Drops one image back to guest resolution for good; later images over the same guest + // memory are created unscaled too, so the decision cannot oscillate. + public void DemoteRenderScale(ResourceSlotIdentifier imageIdentifier) + { + using var held = _lock.Hold(); + if (_slots[imageIdentifier].IsScaled) + { + ReplaceScaled(imageIdentifier); + } + } + + private ResourceSlotIdentifier ReplaceScaled(ResourceSlotIdentifier cachedImageIdentifier) + { + var cached = _slots[cachedImageIdentifier]; + _unscaledRanges.Add(cached.Description.Data); + var replacementImageIdentifier = InsertImage(cached.Description); + var replacement = _slots[replacementImageIdentifier]; + replacement.Uses = cached.Uses; + if (cached.Binding.IsBound || cached.Binding.IsTarget) + { + cached.Binding.NeedsRebind = true; + } + + CopyWholeImage(replacementImageIdentifier, cachedImageIdentifier); + ReleaseImage(cachedImageIdentifier); + return replacementImageIdentifier; + } + private ResourceSlotIdentifier InsertImage(in ImageDescription description) { using var profileScope = RenderPhaseProfile.MeasureDetail(RenderPhaseProfile.Phase.ImageCreate); - var imageIdentifier = _slots.Insert(new CachedImage(_device, _scheduler, _backing, description, _backingPool)); + var requiredBytes = (description.Data.Size + 1023) & ~1023UL; + CollectForAllocation(requiredBytes); + var allowScaling = !IsUnscaledRange(description.Data); + CachedImage image; + try + { + image = new CachedImage(_device, _scheduler, _backing, description, _backingPool, _imageMemoryPool, DescribeImageMemory, allowScaling); + } + catch (CachedImage.ImageOutOfMemoryException) + { + // The budget is an estimate: the device can still run out. Free everything that can + // go, wait for retired images to be destroyed, and try once more. + ReclaimAfterFailedAllocation(); + try + { + image = new CachedImage(_device, _scheduler, _backing, description, _backingPool, _imageMemoryPool, DescribeImageMemory, allowScaling); + } + catch (CachedImage.ImageOutOfMemoryException again) + { + throw SubmissionScheduler.Fatal(again.Message); + } + } + + var imageIdentifier = _slots.Insert(image); if (!ImageDescription.IsEmptyRange(description.Data)) { AddToIndex(imageIdentifier); diff --git a/src/SharpEmu.Libs/Gpu/Images/GuestPixelFormats.cs b/src/SharpEmu.Libs/Gpu/Images/GuestPixelFormats.cs index b730757..862e0a0 100644 --- a/src/SharpEmu.Libs/Gpu/Images/GuestPixelFormats.cs +++ b/src/SharpEmu.Libs/Gpu/Images/GuestPixelFormats.cs @@ -307,8 +307,9 @@ public static class GuestPixelFormats [ new(GuestPixelFormat.Bits8UNorm, 1, 0, 1, true, false), new(GuestPixelFormat.Bits8UScaled, 1, 0, 1, true, false), - new(GuestPixelFormat.Bits8SNorm, 0, 0, 1, false, false), + new(GuestPixelFormat.Bits8SNorm, 1, 0, 1, true, false), new(GuestPixelFormat.Bits8UInt, 1, 0, 1, true, true), + new(GuestPixelFormat.Bits8SInt, 1, 0, 1, true, false, true), new(GuestPixelFormat.Bits16UNorm, 2, 0, 2, true, false), new(GuestPixelFormat.Bits16SNorm, 2, 0, 2, true, false), new(GuestPixelFormat.Bits16UInt, 2, 0, 2, true, true), @@ -429,7 +430,9 @@ public static class GuestPixelFormats private static readonly (GuestPixelFormat Guest, Format Host)[] HostFormats = [ (GuestPixelFormat.Bits8UNorm, Format.R8Unorm), + (GuestPixelFormat.Bits8SNorm, Format.R8SNorm), (GuestPixelFormat.Bits8UInt, Format.R8Uint), + (GuestPixelFormat.Bits8SInt, Format.R8Sint), (GuestPixelFormat.Bits16UNorm, Format.R16Unorm), (GuestPixelFormat.Bits16SNorm, Format.R16SNorm), (GuestPixelFormat.Bits16UInt, Format.R16Uint), diff --git a/src/SharpEmu.Libs/Gpu/Images/ImageBackingPool.cs b/src/SharpEmu.Libs/Gpu/Images/ImageBackingPool.cs index e1a88b3..7ca8957 100644 --- a/src/SharpEmu.Libs/Gpu/Images/ImageBackingPool.cs +++ b/src/SharpEmu.Libs/Gpu/Images/ImageBackingPool.cs @@ -27,7 +27,8 @@ public sealed unsafe class ImageBackingPool : IDisposable ImageCreateFlags Flags, ImageUsageFlags Usage); - private readonly record struct Entry(Image Handle, DeviceMemory Memory, ulong Size); + private readonly record struct Entry(Image Handle, DeviceMemory Memory, ulong Size, + OptimalImageMemoryAllocation? Placement, OptimalImageMemoryPool? Allocator); private readonly GpuDeviceInfo _device; private readonly Dictionary> _entries = new(); @@ -35,6 +36,8 @@ public sealed unsafe class ImageBackingPool : IDisposable private ulong _pooledBytes; private bool _disposed; + internal ulong PooledBytes => _pooledBytes; + public ImageBackingPool(GpuDeviceInfo device) => _device = device; // SHARPEMU_IMAGE_BACKING_POOL=0 destroys every retired image as before. @@ -45,23 +48,30 @@ public sealed unsafe class ImageBackingPool : IDisposable new(create.Format, create.ImageType, create.Extent, create.ArrayLayers, create.MipLevels, create.Samples, create.Flags, create.Usage); public bool TryTake(in Key key, out Image handle, out DeviceMemory memory, out ulong size) + => TryTake(key, out handle, out memory, out size, out _); + + internal bool TryTake(in Key key, out Image handle, out DeviceMemory memory, out ulong size, + out OptimalImageMemoryAllocation? placement) { if (_entries.TryGetValue(key, out var stack) && stack.TryPop(out var entry)) { RemoveOrder(key, entry); _pooledBytes -= entry.Size; (handle, memory, size) = (entry.Handle, entry.Memory, entry.Size); + placement = entry.Placement; return true; } handle = default; memory = default; size = 0; + placement = null; return false; } // False when the pool is full for this key or disposed; the caller then destroys it. - public bool TryReturn(in Key key, Image handle, DeviceMemory memory, ulong size) + public bool TryReturn(in Key key, Image handle, DeviceMemory memory, ulong size, + OptimalImageMemoryAllocation? placement = null, OptimalImageMemoryPool? allocator = null) { if (_disposed || size > MaxPooledBytes) { @@ -79,7 +89,7 @@ public sealed unsafe class ImageBackingPool : IDisposable return false; } - var entry = new Entry(handle, memory, size); + var entry = new Entry(handle, memory, size, placement, allocator); stack.Push(entry); _order.AddLast((key, entry)); _pooledBytes += size; @@ -125,7 +135,8 @@ public sealed unsafe class ImageBackingPool : IDisposable private void Destroy(Entry entry) { _device.Vk.DestroyImage(_device.Device, entry.Handle, null); - _device.FreeMemory(entry.Memory); + if (entry.Placement is { } placement) entry.Allocator!.Free(placement); + else _device.FreeMemory(entry.Memory); } public void Dispose() @@ -136,6 +147,13 @@ public sealed unsafe class ImageBackingPool : IDisposable } _disposed = true; + ReleaseRetained(); + } + + // Entries arrive only after their last GPU submission completes. Trimming + // their storage under pressure does not disable subsequent backing reuse. + public void ReleaseRetained() + { foreach (var (_, entry) in _order) { Destroy(entry); diff --git a/src/SharpEmu.Libs/Gpu/Images/ImageDescription.cs b/src/SharpEmu.Libs/Gpu/Images/ImageDescription.cs index 7c772e6..6e73e83 100644 --- a/src/SharpEmu.Libs/Gpu/Images/ImageDescription.cs +++ b/src/SharpEmu.Libs/Gpu/Images/ImageDescription.cs @@ -89,6 +89,11 @@ public struct ImageDescription public bool Bgra16; public MipLevelLayouts MipLayout; + // The first guest mip the image holds. A streamed texture names only its resident mips + // (base level > 0); the host image starts there, while Extent and MipLayout describe the + // whole guest chain the data is laid out in. + public uint FirstLevel; + public static ImageDescription Create() => new() { HtileClearMask = uint.MaxValue, diff --git a/src/SharpEmu.Libs/Gpu/Images/ImageDropTrace.cs b/src/SharpEmu.Libs/Gpu/Images/ImageDropTrace.cs new file mode 100644 index 0000000..cc59e2a --- /dev/null +++ b/src/SharpEmu.Libs/Gpu/Images/ImageDropTrace.cs @@ -0,0 +1,100 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Runtime.InteropServices; + +namespace SharpEmu.Libs.Gpu.Images; + +// SHARPEMU_TRACE_IMAGE_DROPS=1 reports every time an image loses GPU-written contents that +// never reached guest memory: the code site, the image, how much of it the cause covered and +// the guest render pass (from its push markers). Each surface reports its first occurrences +// and then every power of two, so a surface dropped every frame stays visible but bounded. +internal static class ImageDropTrace +{ + public static readonly bool Enabled = + Environment.GetEnvironmentVariable("SHARPEMU_TRACE_IMAGE_DROPS") == "1"; + + private static readonly Dictionary<(string Site, ulong Address, uint Format, uint Width, uint Height), long> _counts = new(); + private static readonly object _gate = new(); + + // The guest pass that is recording, as nested push-marker names. + public static string Pass { get; private set; } = ""; + + // The byte range whose GPU write invalidates images, while the invalidation runs. + [ThreadStatic] public static ulong CauseAddress; + [ThreadStatic] public static ulong CauseSize; + + public static void Record(CachedImage image, string file, int line) + { + ref readonly var description = ref image.Description; + var site = $"{Path.GetFileNameWithoutExtension(file)}:{line}"; + var key = (site, description.Data.Address, (uint)description.PixelFormat, description.Extent.Width, description.Extent.Height); + long count; + lock (_gate) + { + _counts.TryGetValue(key, out count); + _counts[key] = ++count; + } + + if (count > 3 && (count & (count - 1)) != 0) + { + return; + } + + var coverage = ""; + if (CauseSize != 0) + { + var begin = Math.Max(CauseAddress, description.Data.Address); + var end = Math.Min(CauseAddress + CauseSize, description.Data.Address + description.Data.Size); + var covered = end > begin ? end - begin : 0; + coverage = FormattableString.Invariant( + $" cause=0x{CauseAddress:X}+0x{CauseSize:X} covers={(double)covered / Math.Max(1UL, description.Data.Size):P1}"); + } + + Console.Error.WriteLine( + $"[IMAGE][DROP] site={site} n={count} image=0x{description.Data.Address:X} bytes=0x{description.Data.Size:X} " + + $"{description.Extent.Width}x{description.Extent.Height}x{description.Extent.Depth} format={description.PixelFormat} " + + $"guest={(uint)description.GuestFormat} tile={(uint)description.TileMode} levels={description.Resources.Levels} " + + $"layers={description.Resources.Layers} target={image.Uses.RenderTarget} depth={image.Uses.DepthTarget} " + + $"bufferModified={image.IsBufferModified} cpuDirty={image.IsCpuDirty} " + + $"time={DateTime.Now:HH:mm:ss}{coverage} pass='{Pass}'"); + } + + // One marker stack per command queue: graphics and compute rings interleave their slices. + private static readonly Dictionary> _markers = new(ReferenceEqualityComparer.Instance); + + public static void PushMarker(object queue, ReadOnlySpan payload) + { + var bytes = MemoryMarshal.AsBytes(payload); + var end = bytes.IndexOf((byte)0); + var name = System.Text.Encoding.UTF8.GetString(end < 0 ? bytes : bytes[..end]); + lock (_gate) + { + if (!_markers.TryGetValue(queue, out var stack)) + { + _markers[queue] = stack = new List(); + } + + // A stack that never pops is not a nesting; keep it bounded. + if (stack.Count >= 8) + { + stack.RemoveAt(0); + } + + stack.Add(name); + Pass = string.Join('/', stack); + } + } + + public static void PopMarker(object queue) + { + lock (_gate) + { + if (_markers.TryGetValue(queue, out var stack) && stack.Count != 0) + { + stack.RemoveAt(stack.Count - 1); + Pass = string.Join('/', stack); + } + } + } +} diff --git a/src/SharpEmu.Libs/Gpu/Images/ImageRequest.cs b/src/SharpEmu.Libs/Gpu/Images/ImageRequest.cs index 79be977..beaa5e4 100644 --- a/src/SharpEmu.Libs/Gpu/Images/ImageRequest.cs +++ b/src/SharpEmu.Libs/Gpu/Images/ImageRequest.cs @@ -19,6 +19,9 @@ public struct ImageRequest public ImageViewDescription View; public ImageRole Role; + // The bound program stores into this image, rather than only reading it. + public bool ShaderWrite; + public ImageRequest(in ImageDescription description, in ImageViewDescription view, ImageRole role) { Description = description; diff --git a/src/SharpEmu.Libs/Gpu/Images/ImageRequestBuilders.ColorTarget.cs b/src/SharpEmu.Libs/Gpu/Images/ImageRequestBuilders.ColorTarget.cs index 12bac61..4412e5e 100644 --- a/src/SharpEmu.Libs/Gpu/Images/ImageRequestBuilders.ColorTarget.cs +++ b/src/SharpEmu.Libs/Gpu/Images/ImageRequestBuilders.ColorTarget.cs @@ -41,6 +41,14 @@ public static partial class ImageRequestBuilders { private static long _normalizedSliceRanges; + // Every draw resolves its color targets; the per-level scratch is copied into the + // description before returning, so one cleared pair per thread replaces two arrays per call. + [ThreadStatic] + private static TileLevelSpan[]? _colorTargetMipSpans; + + [ThreadStatic] + private static TilePaddedSize[]? _colorTargetMipPadded; + public static uint SampleCount(uint encodedLog2) => encodedLog2 <= 3 ? 1u << (int)encodedLog2 : 0; private static bool DccAlphaOnMsb(in ColorTargetWords words) @@ -74,6 +82,28 @@ public static partial class ImageRequestBuilders return (supported, fixedSupported, supported ? value : default); } + // The color a fixed DCC clear code decompresses to; the register code (0x20) is not fixed. + public static bool TryFixedDccClearValue(byte code, out ClearColorValue value) + { + value = default; + switch (code) + { + case 0x00: + return true; + case 0x40: + value.Float32_3 = 1f; + return true; + case 0x80: + value.Float32_0 = value.Float32_1 = value.Float32_2 = 1f; + return true; + case 0xc0: + value.Float32_0 = value.Float32_1 = value.Float32_2 = value.Float32_3 = 1f; + return true; + default: + return false; + } + } + public static bool SupportsDccFixedClear(Format format) => PackedClearValue.SupportsDccFixedColor(format); // Builds the request for a bound color target. Null when the slot carries no target. @@ -235,8 +265,10 @@ public static partial class ImageRequestBuilders pitch = TileGeometry.TexturePitch(transferFormat, width, GuestTileMode.Linear); } - var mipSpans = new TileLevelSpan[TiledSurfaceLayout.MaxLevels]; - var mipPadded = new TilePaddedSize[TiledSurfaceLayout.MaxLevels]; + var mipSpans = _colorTargetMipSpans ??= new TileLevelSpan[TiledSurfaceLayout.MaxLevels]; + var mipPadded = _colorTargetMipPadded ??= new TilePaddedSize[TiledSurfaceLayout.MaxLevels]; + Array.Clear(mipSpans); + Array.Clear(mipPadded); TiledSurfaceLayout? volumeLayout = null; ulong size; ulong backingSize = 0; diff --git a/src/SharpEmu.Libs/Gpu/Images/ImageRequestBuilders.DepthTarget.cs b/src/SharpEmu.Libs/Gpu/Images/ImageRequestBuilders.DepthTarget.cs index 8a0c357..d61b7cd 100644 --- a/src/SharpEmu.Libs/Gpu/Images/ImageRequestBuilders.DepthTarget.cs +++ b/src/SharpEmu.Libs/Gpu/Images/ImageRequestBuilders.DepthTarget.cs @@ -71,6 +71,17 @@ public static partial class ImageRequestBuilders var hasStencil = depthWords.StencilFormat != GuestStencilFormat.Invalid; if (!copyMode) { + // With both formats INVALID the DB has no depth or stencil surface: depth and stencil + // tests and writes do not happen, whatever DB_DEPTH_CONTROL and the other fields say. + if (depthWords.DepthFormat == GuestDepthFormat.Invalid && !hasStencil) + { + if (Rendering.RenderTrace.Enabled) + { + Rendering.RenderTrace.Write("DepthAttachmentRejected reason=invalid-formats"); + } + return null; + } + var depthActive = depthWords.DepthTestEnabled || depthWords.DepthWriteEnabled || depthWords.DepthBoundsEnabled || depthWords.DepthClearEnabled || depthWords.CopyDepthToColor; var stencilActive = hasStencil && (depthWords.StencilTestEnabled || depthWords.StencilClearEnabled || depthWords.CopyStencilToColor); if (!depthActive && !stencilActive) @@ -144,9 +155,9 @@ public static partial class ImageRequestBuilders throw SubmissionScheduler.Fatal($"The HTile metadata address is invalid: htile=0x{depthWords.HtileBase:X16}."); } - if (depthWords.SliceMax >= 32) + if (depthWords.SliceMax >= SurfaceMetadata.MaxSlices) { - throw SubmissionScheduler.Fatal($"HTile clear tracking supports at most 32 slices: last={depthWords.SliceMax}."); + throw SubmissionScheduler.Fatal($"HTile clear tracking supports at most {SurfaceMetadata.MaxSlices} slices: last={depthWords.SliceMax}."); } } diff --git a/src/SharpEmu.Libs/Gpu/Images/ImageRequestBuilders.Texture.cs b/src/SharpEmu.Libs/Gpu/Images/ImageRequestBuilders.Texture.cs index 1508dd0..ab4cadd 100644 --- a/src/SharpEmu.Libs/Gpu/Images/ImageRequestBuilders.Texture.cs +++ b/src/SharpEmu.Libs/Gpu/Images/ImageRequestBuilders.Texture.cs @@ -39,7 +39,7 @@ public static partial class ImageRequestBuilders ? GuestImageType.Color3D : shape.OneDimensional ? GuestImageType.Color1D : GuestImageType.Color2D; description.Extent = new Extent3D(1, 1, 1); - description.Resources = SubresourceCount.Single; + description.Resources = new SubresourceCount(1, shape.Cube ? 6u : 1u); description.BytesPerBlock = 4; description.Samples = shape.Multisampled ? 4u : 1u; description.MipLayout[0] = new MipLevelLayout { Offset = 0, Size = 0, Pitch = 1, Height = 1 }; @@ -48,17 +48,20 @@ public static partial class ImageRequestBuilders Format = format, Type = shape.Volume ? ImageViewType.Type3D + : shape.Cube + ? ImageViewType.TypeCube : shape.OneDimensional ? shape.Arrayed ? ImageViewType.Type1DArray : ImageViewType.Type1D : shape.Arrayed ? ImageViewType.Type2DArray : ImageViewType.Type2D, Aspect = depthCompare ? ImageAspectFlags.DepthBit : ImageAspectFlags.ColorBit, + LayerCount = shape.Cube ? 6u : 1u, Usage = storage ? ImageUsageFlags.StorageBit : ImageUsageFlags.SampledBit, }; return new ImageRequest(description, view, storage ? ImageRole.StorageImage : ImageRole.Texture); } // A depth placeholder reads its value into red, as sampled depth views do. - private static TextureRequestResolution NullTextureResolution(in ShaderImageShape shape) + public static TextureRequestResolution NullTextureResolution(in ShaderImageShape shape) { var request = NullTexture(shape); var swizzle = request.Description.IsDepth ? ViewFormatRules.PackDestinationSelect(4, 0, 0, 1) : 0u; @@ -112,13 +115,15 @@ public static partial class ImageRequestBuilders // The view follows the compiled module: a volume, a layer window to the last layer, or one layer. private static ImageViewDescription TextureView( - in TextureDescriptorWords descriptor, in ShaderImageShape shape, Format format, bool shaderConversion, uint viewLevels, uint imageLayers) + in TextureDescriptorWords descriptor, in ShaderImageShape shape, Format format, bool shaderConversion, uint viewLevels, uint imageLayers, + uint firstLevel = 0) { + var baseLevel = descriptor.BaseLevel - firstLevel; var mapping = shape.Storage || shaderConversion ? default : ViewFormatRules.ComponentMapping(DestinationSwizzle(descriptor)); var usage = shape.Storage ? ImageUsageFlags.StorageBit : ImageUsageFlags.SampledBit; if (shape.Volume) { - return WithMinLod(new ImageViewDescription(format, ImageViewType.Type3D, ImageAspectFlags.ColorBit, descriptor.BaseLevel, viewLevels, 0, 1, mapping, usage), descriptor, shape); + return WithMinLod(new ImageViewDescription(format, ImageViewType.Type3D, ImageAspectFlags.ColorBit, baseLevel, viewLevels, 0, 1, mapping, usage), descriptor, shape, firstLevel); } var baseLayer = descriptor.BaseArray; @@ -131,15 +136,16 @@ public static partial class ImageRequestBuilders var type = shape.OneDimensional ? shape.Arrayed ? ImageViewType.Type1DArray : ImageViewType.Type1D : shape.Arrayed ? ImageViewType.Type2DArray : ImageViewType.Type2D; - return WithMinLod(new ImageViewDescription(format, type, ImageAspectFlags.ColorBit, descriptor.BaseLevel, viewLevels, baseLayer, layerCount, mapping, usage), descriptor, shape); + return WithMinLod(new ImageViewDescription(format, type, ImageAspectFlags.ColorBit, baseLevel, viewLevels, baseLayer, layerCount, mapping, usage), descriptor, shape, firstLevel); } // MIN_LOD (4.8 fixed point, absolute mip levels) keeps sampling off mips a streamed texture has not // loaded yet; ignoring it sampled the unloaded mip 0 of Astro Bot's stadium sky and bloom spread its // garbage HDR values over the screen. Storage views address levels explicitly and are not clamped. - private static ImageViewDescription WithMinLod(ImageViewDescription view, in TextureDescriptorWords descriptor, in ShaderImageShape shape) + private static ImageViewDescription WithMinLod(ImageViewDescription view, in TextureDescriptorWords descriptor, in ShaderImageShape shape, uint firstLevel) { - var minLod = descriptor.MinLod / 256f; + // The host image starts at the first resident guest level; view levels count from there. + var minLod = descriptor.MinLod / 256f - firstLevel; if (shape.Storage || minLod <= view.BaseLevel) { return view; @@ -148,6 +154,27 @@ public static partial class ImageRequestBuilders return view with { MinLod = Math.Min(minLod, view.BaseLevel + view.LevelCount - 1) }; } + // The same texture as an image of the whole guest mip chain, viewed from the descriptor's + // base level: the request a texture gets when its larger mips are mapped. + public static ImageRequest WithWholeMipChain(in ImageRequest request) + { + var firstLevel = request.Description.FirstLevel; + if (firstLevel == 0) + { + return request; + } + + var whole = request; + whole.Description.Resources = new SubresourceCount(request.Description.Resources.Levels + firstLevel, request.Description.Resources.Layers); + whole.Description.FirstLevel = 0; + whole.View = request.View with + { + BaseLevel = request.View.BaseLevel + firstLevel, + MinLod = request.View.MinLod == 0 ? 0 : request.View.MinLod + firstLevel, + }; + return whole; + } + public static uint DestinationSwizzle(in TextureDescriptorWords descriptor) => ViewFormatRules.PackDestinationSelect(descriptor.DestinationSelectX, descriptor.DestinationSelectY, descriptor.DestinationSelectZ, descriptor.DestinationSelectW); @@ -173,6 +200,20 @@ public static partial class ImageRequestBuilders [ThreadStatic] private static Dictionary? _textureCache; + // Whether the T# describes a texture the host can create: a non-null base and a format with + // a host equivalent. The hardware reads any other descriptor as the null texture. + public static bool DescribesHostTexture(ReadOnlySpan words) + { + if (words.Length < 4) + { + return false; + } + + var descriptor = new TextureDescriptorWords(words); + return !descriptor.IsNull && + GuestPixelFormats.HostFormat(GuestPixelFormats.RemapTextureFormat(descriptor.Format)) != Format.Undefined; + } + // Builds the request for a sampled or storage texture; the words are the eight T# dwords. public static TextureRequestResolution Texture(ReadOnlySpan words, in ShaderImageShape shape) { @@ -185,7 +226,7 @@ public static partial class ImageRequestBuilders return cached; } - var resolution = BuildTexture(key, shape); + var resolution = BuildTexture(key, shape, tolerateInvalid: false, out _); if (cache.Count >= TextureCacheLimit) { cache.Clear(); @@ -195,8 +236,49 @@ public static partial class ImageRequestBuilders return resolution; } - private static TextureRequestResolution BuildTexture(ReadOnlySpan words, in ShaderImageShape shape) + // A T# read at run time can be any lane's register contents, which the hardware reads as + // the null texture when they describe no valid view. False instead of stopping on it. + public static bool TryTexture(ReadOnlySpan words, in ShaderImageShape shape, out TextureRequestResolution resolution) { + Span key = stackalloc uint[8]; + words[..Math.Min(words.Length, 8)].CopyTo(key); + var cacheKey = new TextureKey(key[0], key[1], key[2], key[3], key[4], key[5], key[6], key[7], shape); + var cache = _textureCache ??= new Dictionary(); + if (cache.TryGetValue(cacheKey, out resolution)) + { + return true; + } + + resolution = BuildTexture(key, shape, tolerateInvalid: true, out var invalid); + if (invalid is not null) + { + return false; + } + + if (cache.Count >= TextureCacheLimit) + { + cache.Clear(); + } + + cache[cacheKey] = resolution; + return true; + } + + private static TextureRequestResolution Reject(string message, bool tolerateInvalid, out string? invalid) + { + if (!tolerateInvalid) + { + throw SubmissionScheduler.Fatal(message); + } + + invalid = message; + return default; + } + + private static TextureRequestResolution BuildTexture(ReadOnlySpan words, in ShaderImageShape shape, bool tolerateInvalid, + out string? invalid) + { + invalid = null; Span padded = stackalloc uint[8]; words[..Math.Min(words.Length, 8)].CopyTo(padded); var descriptor = new TextureDescriptorWords(padded); @@ -211,6 +293,11 @@ public static partial class ImageRequestBuilders var height = descriptor.Height + 1; var baseLevel = descriptor.BaseLevel; var lastLevel = descriptor.LastLevel; + if (descriptor.Type is < GuestImageType.Color1D or > GuestImageType.Color2DMsaaArray) + { + return Reject($"The texture type is not an image type: address=0x{address:X16} type={(uint)descriptor.Type}.", tolerateInvalid, out invalid); + } + var type = TextureType(descriptor.Type); var multisampled = IsMultisampledTexture(type); var maxMip = shape.R128 ? lastLevel : descriptor.MaxMip; @@ -232,9 +319,9 @@ public static partial class ImageRequestBuilders (baseLevel != 0 || lastLevel == 0 || lastLevel > 3 || maxMip != lastLevel || !msaaTile || (descriptor.MsaaDepth && !depthTile) || (!msaaArray && (descriptor.Depth != 0 || descriptor.BaseArray != 0))))) { - throw SubmissionScheduler.Fatal( + return Reject( $"The texture mip view is not supported: address=0x{address:X16} type={(uint)type} baseLevel={baseLevel} lastLevel={lastLevel} maxMip={maxMip} " + - $"levels={levels} tile={(uint)tile} depth={descriptor.Depth} baseArray={descriptor.BaseArray} msaaDepth={descriptor.MsaaDepth} storage={storage}."); + $"levels={levels} tile={(uint)tile} depth={descriptor.Depth} baseArray={descriptor.BaseArray} msaaDepth={descriptor.MsaaDepth} storage={storage}.", tolerateInvalid, out invalid); } var samples = multisampled ? 1u << (int)lastLevel : 1u; @@ -250,8 +337,8 @@ public static partial class ImageRequestBuilders (volume || multisampled || width != height || descriptor.BaseArray > descriptor.Depth || (descriptor.Depth - descriptor.BaseArray + 1) % 6 != 0)) { - throw SubmissionScheduler.Fatal( - $"The cubemap view is invalid: address=0x{address:X16} extent={width}x{height} layers={imageLayers} baseArray={descriptor.BaseArray} samples={samples}."); + return Reject( + $"The cubemap view is invalid: address=0x{address:X16} extent={width}x{height} layers={imageLayers} baseArray={descriptor.BaseArray} samples={samples}.", tolerateInvalid, out invalid); } uint pitch; TileSizeAndAlignment size; @@ -261,8 +348,8 @@ public static partial class ImageRequestBuilders pitch = depthTile ? TileGeometry.DepthPitch(width, bytes, lastLevel) : TileGeometry.RenderTargetPitch(width, bytes, lastLevel); if (pitch == 0 || !TileGeometry.TryGetRenderTargetSize(width, height, pitch, bytes, out size, lastLevel) || size.Size > uint.MaxValue / imageLayers) { - throw SubmissionScheduler.Fatal( - $"The multisample texture layout is not supported: address=0x{address:X16} extent={width}x{height} bytes={bytes} pitch={pitch} samplesLog2={lastLevel} layers={imageLayers}."); + return Reject( + $"The multisample texture layout is not supported: address=0x{address:X16} extent={width}x{height} bytes={bytes} pitch={pitch} samplesLog2={lastLevel} layers={imageLayers}.", tolerateInvalid, out invalid); } size = new TileSizeAndAlignment(size.Size * imageLayers, size.Align); @@ -273,10 +360,12 @@ public static partial class ImageRequestBuilders size = TileGeometry.TextureTotalSize(format, width, height, volume ? depth : imageLayers, levels, tile, volume); } - if (size.Size == 0 || size.Align == 0 || (address & (size.Align - 1UL)) != 0) + // T# encodes its byte address in 256-byte units. The layout alignment + // describes a new surface allocation, not an addressed view inside one. + if (size.Size == 0 || size.Align == 0 || (address & 0xFFUL) != 0) { - throw SubmissionScheduler.Fatal( - $"The texture footprint or alignment is invalid: address=0x{address:X16} size=0x{size.Size:X} align=0x{size.Align:X} format={(uint)format} tile={(uint)tile}."); + return Reject( + $"The texture footprint or alignment is invalid: address=0x{address:X16} size=0x{size.Size:X} align=0x{size.Align:X} format={(uint)format} tile={(uint)tile}.", tolerateInvalid, out invalid); } var pixelFormat = surfaceFormat.HostFormat; @@ -286,7 +375,11 @@ public static partial class ImageRequestBuilders pixelFormat = depthFormat.DepthAttachmentFormat; } // Atomic storage images are declared as UINT in SPIR-V, including float atomics. - var storageViewFormat = storage && (shape.Atomic || format == GuestPixelFormat.Bits32SInt) ? Format.R32Uint : ViewFormatRules.SrgbStorageFormat(pixelFormat); + // Write-only SINT storage images are declared as UINT and use the same-size UINT view. + var storageViewFormat = !storage ? Format.Undefined + : shape.Atomic || format == GuestPixelFormat.Bits32SInt ? Format.R32Uint + : shape.NumericClass == TextureNumericClass.Uint && ViewFormatRules.UintStorageFormat(pixelFormat) is var uintView && uintView != Format.Undefined ? uintView + : ViewFormatRules.SrgbStorageFormat(pixelFormat); var viewFormat = storage && storageViewFormat != Format.Undefined ? storageViewFormat : pixelFormat; var blockBytes = GuestPixelFormats.BlockCompressedBytes(format); var description = ImageDescription.Create(); @@ -317,10 +410,35 @@ public static partial class ImageRequestBuilders } else { + if (description.IsVolume && description.TileMode != GuestTileMode.Linear && + !TileGeometry.TryGetTiledTextureLayout(new TiledSurfaceDescription(description.GuestFormat, description.TileMode, + TileSurfaceDimension.Volume3D, width, height, depth, levels, 1), out _)) + { + return Reject($"The volume texture layout is not supported: address=0x{address:X16} format={(uint)format} tile={(uint)tile} " + + $"extent={width}x{height}x{depth} levels={levels}.", tolerateInvalid, out invalid); + } + PopulateTextureMipLayout(ref description); } - var view = TextureView(descriptor, shape, viewFormat, shaderConversion, viewLevels, description.Resources.Layers); + // A sampled texture whose descriptor starts past mip 0 may be a streamed texture whose + // larger mips are not mapped: the request then holds only the levels from there. The + // image cache returns it to the whole chain when those mips are mapped after all + // (WithWholeMipChain), as a view of a lower mip of a rendered image is. + var firstLevel = !storage && !volume && samples == 1 && baseLevel > 0 && baseLevel < levels ? baseLevel : 0u; + if (firstLevel != 0) + { + description.FirstLevel = firstLevel; + description.Resources = new SubresourceCount(levels - firstLevel, imageLayers); + } + + if (!shape.Volume && descriptor.BaseArray >= description.Resources.Layers) + { + return Reject($"The texture base layer is outside the image: baseLayer={descriptor.BaseArray} layers={description.Resources.Layers} address=0x{address:X16}.", + tolerateInvalid, out invalid); + } + + var view = TextureView(descriptor, shape, viewFormat, shaderConversion, viewLevels, description.Resources.Layers, firstLevel); var request = new ImageRequest(description, view, storage ? ImageRole.StorageImage : ImageRole.Texture); return new TextureRequestResolution(request, shaderConversion, pixelFormat, DestinationSwizzle(descriptor)); } diff --git a/src/SharpEmu.Libs/Gpu/Images/RenderScalePolicy.cs b/src/SharpEmu.Libs/Gpu/Images/RenderScalePolicy.cs new file mode 100644 index 0000000..ab85857 --- /dev/null +++ b/src/SharpEmu.Libs/Gpu/Images/RenderScalePolicy.cs @@ -0,0 +1,146 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Globalization; +using Silk.NET.Vulkan; + +namespace SharpEmu.Libs.Gpu.Images; + +// Picks the host resolution of one guest image. +// +// The choice is a pure function of the guest description, so every view, alias, copy and +// descriptor of the same guest memory reaches the same answer without consulting the cache. +// Guest memory layouts never change: a scaled image only differs in the size of its host +// backing, and the transfer paths route guest bytes through a guest-sized intermediate. +public static class RenderScalePolicy +{ + public const string VariableName = "SHARPEMU_RENDER_SCALE"; + + // Below this a target is a shadow map, a LUT, a probe atlas or an eye-adaptation + // readback, and scaling it changes what the game computes rather than how it looks. + public const uint MinimumScaledWidth = 1280; + public const uint MinimumScaledHeight = 720; + + private static readonly float? _environmentScale = ReadEnvironmentScale(); + private static float _scale = _environmentScale ?? 1.0f; + + public static float Scale => Volatile.Read(ref _scale); + + public static bool Enabled => Scale != 1.0f; + + // The environment variable wins so scripted runs do not depend on stored settings. + public static void Configure(float scale) + { + if (_environmentScale is not null) + { + return; + } + + Volatile.Write(ref _scale, VideoOut.HostVideoOptions.NormalizeRenderScale(scale)); + } + + private static float? ReadEnvironmentScale() + { + var text = Environment.GetEnvironmentVariable(VariableName); + if (string.IsNullOrWhiteSpace(text) || + !float.TryParse(text, NumberStyles.Float, CultureInfo.InvariantCulture, out var parsed)) + { + return null; + } + + return VideoOut.HostVideoOptions.NormalizeRenderScale(parsed); + } + + public static uint ScaleLength(uint length, float scale) + { + if (scale == 1.0f || length <= 1) + { + return length; + } + + var scaled = (uint)Math.Ceiling(length * (double)scale); + return Math.Clamp(scaled, 1u, 32768u); + } + + // The guest-pixel extent the clip-disable vertex transform maps onto. Dividing the device + // limit by an upscale keeps the host viewport (reference x scale) inside that limit while + // the transform the vertex program bakes stays scale independent. + public static uint ClipSpaceReferenceExtent(uint limit) + { + var scale = Scale; + return scale <= 1.0f ? limit : Math.Max(1u, (uint)(limit / scale)); + } + + public static Extent3D ScaleExtent(Extent3D extent, float scale) => scale == 1.0f + ? extent + : new Extent3D(ScaleLength(extent.Width, scale), ScaleLength(extent.Height, scale), extent.Depth); + + private static Func? _formatSupport; + + // Installed once with the device: a scaled image is resized with vkCmdBlitImage whenever + // guest bytes or a differently scaled alias cross into it, so a format that cannot blit + // must keep its guest resolution. + public static void ConfigureFormatSupport(Func supportsScaledResize) => + Volatile.Write(ref _formatSupport, supportsScaledResize); + + public static float ScaleFor(in ImageDescription description) => ScaleFor(description, Scale); + + // Only screen-class, single-level, single-sample, uncompressed 2D render and depth + // targets scale. Everything the host cannot reproduce at another size - block grids, + // multisample footprints, HTile and DCC block coverage, mip chains whose levels would + // round apart, cube and array faces - keeps its guest resolution. + public static float ScaleFor(in ImageDescription description, float scale) + { + if (scale == 1.0f || !IsScalable(description)) + { + return 1.0f; + } + + return scale; + } + + public static bool IsScalable(in ImageDescription description) + { + // A stencil-association entry carries no host image, and an image without guest + // memory is an internal helper whose extent must track whatever it mirrors. + if (description.PixelFormat == Format.Undefined || !ImageDescription.IsValidRange(description.Data)) + { + return false; + } + + if (description.Type != GuestImageType.Color2D || description.Samples != 1 || + description.Resources.Levels != 1 || description.Resources.Layers != 1) + { + return false; + } + + // DCC only describes guest bytes the host never stores, so a scaled backing keeps it + // meaningful. HTile encodes guest depth blocks the renderer reads back as clear state. + if (description.IsBlock || description.HasStencil || description.Metadata.Kind == MetadataKind.Htile) + { + return false; + } + + // Render and depth tilings mark the surfaces a game draws into. Asset textures use + // the standard tilings and must stay at their authored size. + if (description.TileMode is not (GuestTileMode.RenderTarget or GuestTileMode.Depth)) + { + return false; + } + + if (description.Extent.Width < MinimumScaledWidth || description.Extent.Height < MinimumScaledHeight) + { + return false; + } + + // A square target at screen size is a cube face or a shadow atlas far more often + // than it is a scene buffer. + if (description.Extent.Width == description.Extent.Height) + { + return false; + } + + // Only a backend that installed the predicate (the Vulkan device) can resize an image. + return Volatile.Read(ref _formatSupport) is { } supported && supported(description.PixelFormat); + } +} diff --git a/src/SharpEmu.Libs/Gpu/Images/SamplerStore.cs b/src/SharpEmu.Libs/Gpu/Images/SamplerStore.cs index 466023e..5500516 100644 --- a/src/SharpEmu.Libs/Gpu/Images/SamplerStore.cs +++ b/src/SharpEmu.Libs/Gpu/Images/SamplerStore.cs @@ -1,6 +1,7 @@ // Copyright (C) 2026 SharpEmu Emulator Project // SPDX-License-Identifier: GPL-2.0-or-later +using System.Collections.Concurrent; using SharpEmu.Libs.Gpu.Buffers; using SharpEmu.Libs.Gpu.Scheduling; using Silk.NET.Vulkan; @@ -48,7 +49,9 @@ public enum SamplerBorderColor : uint public sealed unsafe class SamplerStore : IDisposable { private readonly GpuDeviceInfo _device; - private readonly Dictionary<(uint, uint, uint, uint, bool), Sampler> _samplers = new(); + // Lookups run for every sampler of every draw on more than one queue thread; only a + // miss takes the lock, so concurrent draws never wait on each other for a known sampler. + private readonly ConcurrentDictionary<(uint, uint, uint, uint, bool), Sampler> _samplers = new(); private readonly object _gate = new(); public SamplerStore(GpuDeviceInfo device) => _device = device; @@ -57,16 +60,21 @@ public sealed unsafe class SamplerStore : IDisposable public Sampler GetSampler(in SamplerDescriptorWords words, bool integerView) { + var key = (words[0], words[1], words[2], words[3], integerView); + if (_samplers.TryGetValue(key, out var existing)) + { + return existing; + } + lock (_gate) { - var key = (words[0], words[1], words[2], words[3], integerView); - if (_samplers.TryGetValue(key, out var existing)) + if (_samplers.TryGetValue(key, out existing)) { return existing; } var sampler = CreateSampler(words, integerView); - _samplers.Add(key, sampler); + _samplers[key] = sampler; return sampler; } } diff --git a/src/SharpEmu.Libs/Gpu/Images/SurfaceMetadata.cs b/src/SharpEmu.Libs/Gpu/Images/SurfaceMetadata.cs index f547685..9566335 100644 --- a/src/SharpEmu.Libs/Gpu/Images/SurfaceMetadata.cs +++ b/src/SharpEmu.Libs/Gpu/Images/SurfaceMetadata.cs @@ -16,9 +16,40 @@ public enum SurfaceMetadataKind : byte public sealed class SurfaceMetadata { + // Slices 0-31 live in ClearMask; an array target can hold up to 2048 slices. + public const uint MaxSlices = 2048; + public SurfaceMetadataKind Kind; public uint ClearMask; public uint FillValue = 0xffffffff; public ulong FillSize; + private System.Collections.BitArray? _upperSlices; + + // Sets slices 0-31 from the mask; the rest follow an all-clear or all-uncleared mask. + public void SetFromMask(uint mask) + { + ClearMask = mask; + if (mask == uint.MaxValue) + (_upperSlices ??= new System.Collections.BitArray((int)(MaxSlices - 32))).SetAll(true); + else + _upperSlices?.SetAll(false); + } + + public bool IsSliceCleared(uint slice) => + slice < 32 ? (ClearMask & (1u << (int)slice)) != 0 : slice < MaxSlices && _upperSlices is { } upper && upper[(int)(slice - 32)]; + + public bool SetSlice(uint slice, bool isClear) + { + if (slice >= MaxSlices) + return false; + if (slice < 32) + { + ClearMask = isClear ? ClearMask | (1u << (int)slice) : ClearMask & ~(1u << (int)slice); + return true; + } + + (_upperSlices ??= new System.Collections.BitArray((int)(MaxSlices - 32)))[(int)(slice - 32)] = isClear; + return true; + } public ulong Size; } diff --git a/src/SharpEmu.Libs/Gpu/Images/TextureTransferLayout.cs b/src/SharpEmu.Libs/Gpu/Images/TextureTransferLayout.cs index 989e5e3..4cbdea5 100644 --- a/src/SharpEmu.Libs/Gpu/Images/TextureTransferLayout.cs +++ b/src/SharpEmu.Libs/Gpu/Images/TextureTransferLayout.cs @@ -183,7 +183,8 @@ public sealed class TextureTransferLayout ulong uploadSize, bool allowDepthTile, bool volume, - string owner) + string owner, + ReadOnlySpan guestLinearLevels = default) { var layout = new TextureTransferLayout(); var description = new TiledSurfaceDescription( @@ -231,7 +232,23 @@ public sealed class TextureTransferLayout } layout.Pitch = TileGeometry.TexturePitch(format, width, tile); - if (tile == GuestTileMode.Linear) + if (tile == GuestTileMode.Linear && guestLinearLevels.Length >= levels) + { + // A linear target's builder already laid its levels out with the guest's pitch + // (a render target's need not be aligned like a texture's); copy that layout. + for (uint level = 0; level < levels; level++) + { + var guest = guestLinearLevels[(int)level]; + layout.Mips[level] = new TransferMipLayout + { + Offset = guest.Offset, + Size = guest.Size, + RowLength = guest.Pitch, + ImageHeight = guest.Height, + }; + } + } + else if (tile == GuestTileMode.Linear) { var levelSpans = new TileLevelSpan[TiledSurfaceLayout.MaxLevels]; var paddedSizes = new TilePaddedSize[TiledSurfaceLayout.MaxLevels]; @@ -273,6 +290,31 @@ public sealed class TextureTransferLayout private static Exception UnsupportedTiledUpload(string owner, GuestPixelFormat format, GuestTileMode tile, ulong uploadSize, uint width, uint height, uint levels) => SubmissionScheduler.Fatal($"{owner}: the tiled texture upload is not supported: format={(uint)format} tile={(uint)tile} size={uploadSize} extent={width}x{height} levels={levels}."); + // The last byte a copy reaches in its buffer, plus one, following Vulkan's buffer + // addressing: rows advance by BufferRowLength and slices by BufferImageHeight. + public static ulong CopyFootprint(IReadOnlyList regions, TileElementLayout element) + { + static ulong Blocks(uint texels, uint block) => (texels + (ulong)block - 1) / block; + ulong end = 0; + foreach (var region in regions) + { + var extent = region.ImageExtent; + if (extent.Width == 0 || extent.Height == 0 || extent.Depth == 0 || region.ImageSubresource.LayerCount == 0) + { + continue; + } + + var rowBlocks = Blocks(region.BufferRowLength == 0 ? extent.Width : region.BufferRowLength, element.TexelWidth); + var sliceRows = Blocks(region.BufferImageHeight == 0 ? extent.Height : region.BufferImageHeight, element.TexelHeight); + var slices = (ulong)extent.Depth * region.ImageSubresource.LayerCount; + var lastBlock = ((slices - 1) * sliceRows + Blocks(extent.Height, element.TexelHeight) - 1) * rowBlocks + + Blocks(extent.Width, element.TexelWidth); + end = Math.Max(end, region.BufferOffset + lastBlock * element.Bytes); + } + + return end; + } + public List BuildCopies() { var description = Surface.Description; diff --git a/src/SharpEmu.Libs/Gpu/Images/ViewFormatRules.cs b/src/SharpEmu.Libs/Gpu/Images/ViewFormatRules.cs index adc30b6..e5b1c2f 100644 --- a/src/SharpEmu.Libs/Gpu/Images/ViewFormatRules.cs +++ b/src/SharpEmu.Libs/Gpu/Images/ViewFormatRules.cs @@ -272,6 +272,21 @@ public static class ViewFormatRules _ => Format.Undefined, }; + // The same-size UINT view of a SINT format; integer stores write identical bits through it. + public static Format UintStorageFormat(Format imageFormat) => imageFormat switch + { + Format.R8Sint => Format.R8Uint, + Format.R8G8Sint => Format.R8G8Uint, + Format.R8G8B8A8Sint => Format.R8G8B8A8Uint, + Format.R16Sint => Format.R16Uint, + Format.R16G16Sint => Format.R16G16Uint, + Format.R16G16B16A16Sint => Format.R16G16B16A16Uint, + Format.R32Sint => Format.R32Uint, + Format.R32G32Sint => Format.R32G32Uint, + Format.R32G32B32A32Sint => Format.R32G32B32A32Uint, + _ => Format.Undefined, + }; + public static bool IsSupportedSampledColorView(Format imageFormat, Format viewFormat, uint swizzle) => IsValidSwizzle(swizzle) && AreCompatible(imageFormat, viewFormat); diff --git a/src/SharpEmu.Libs/Gpu/Metal/MetalCommandStreamHost.Rendering.cs b/src/SharpEmu.Libs/Gpu/Metal/MetalCommandStreamHost.Rendering.cs index 1e5b709..53b2726 100644 --- a/src/SharpEmu.Libs/Gpu/Metal/MetalCommandStreamHost.Rendering.cs +++ b/src/SharpEmu.Libs/Gpu/Metal/MetalCommandStreamHost.Rendering.cs @@ -166,6 +166,8 @@ internal sealed partial class MetalCommandStreamHost : IRenderHost, IShaderPipel // Metal has no 64-bit workgroup atomics here; keep the non-atomic 32-bit pair. bool IShaderPipelineHost.SharedInt64AtomicsEnabled => false; + // The Metal backend runs no f16 conversion probe, so it keeps the exact emulation. + bool IShaderPipelineHost.NativeHalfConversionExact => false; RenderHostLimits IShaderPipelineHost.Limits => new(MaxDimension, MaxDimension, MaxDimension, MaxDimension); @@ -236,6 +238,8 @@ internal sealed partial class MetalCommandStreamHost : IRenderHost, IShaderPipel void IRenderHost.RunPendingOperations() => RunPendingCommands(); + bool IRenderHost.TryReadCleanGuestBytes(ulong address, Span destination) => false; + void IRenderHost.SetDebugInformation(RecordedOperation operation, ulong submitId, uint argument0, uint argument1, uint argument2, uint argument3, ulong argument4) { _ = (operation, submitId, argument0, argument1, argument2, argument3, argument4); @@ -278,6 +282,18 @@ internal sealed partial class MetalCommandStreamHost : IRenderHost, IShaderPipel _ = image; } + // The Metal backend never scales an image, so its attachments always agree. + void IRenderHost.DemoteRenderScale(ResourceSlotIdentifier image) + { + _ = image; + } + + float IRenderHost.GetRenderScale(ResourceSlotIdentifier image) + { + _ = image; + return 1f; + } + // The records of one draw or dispatch live until the executor resets; the submitted copy owns its bytes. void IRenderHost.ResetBindings() { @@ -312,6 +328,9 @@ internal sealed partial class MetalCommandStreamHost : IRenderHost, IShaderPipel BufferBinding IRenderHost.NullBuffer => new(0, 0); + // Metal bindings retain guest ranges directly; acquiring another range never replaces a handle. + void IRenderHost.PrepareBufferAllocations(ReadOnlySpan ranges) { } + BufferBinding IRenderHost.ObtainBuffer(ulong address, ulong size, bool isWritten) { _ = isWritten; diff --git a/src/SharpEmu.Libs/Gpu/Pipelines/BindlessImageHeap.cs b/src/SharpEmu.Libs/Gpu/Pipelines/BindlessImageHeap.cs new file mode 100644 index 0000000..06fbff5 --- /dev/null +++ b/src/SharpEmu.Libs/Gpu/Pipelines/BindlessImageHeap.cs @@ -0,0 +1,414 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.Libs.Gpu.Buffers; +using SharpEmu.Libs.Gpu.Scheduling; +using SharpEmu.ShaderCompiler.Resources; +using Silk.NET.Vulkan; + +namespace SharpEmu.Libs.Gpu.Pipelines; + +// One update-after-bind image set shared by all translated guest pipelines. The +// shader still has separate Vulkan image types for dimensions and numeric classes, +// but the arrays themselves are runtime arrays owned by this set rather than by a +// shader permutation. +public sealed unsafe class BindlessImageHeap : IDisposable +{ + // Sampled images, storage images, and the samplers runtime-descriptor accesses index + // (BindingLayout.BindlessSamplerBinding). + private const uint BindingCount = 3; + private const uint SamplerBinding = BindingLayout.BindlessSamplerBinding; + private const uint SamplerCapacity = 4096; + // NVIDIA reports maxUpdateAfterBindDescriptorsInAllPools as UINT_MAX. Keep + // the persistent layout bounded even when that device-wide limit is not + // useful, while retaining the full capacity on devices with lower limits. + private const uint StableCapacityPerBinding = 128u * 1024u; + + private readonly struct SlotKey : IEquatable + { + private const int MaxWords = 8; + + public readonly DescriptorBindingKind Kind; + public readonly ulong View; + public readonly ImageLayout Layout; + private readonly uint _wordCount; + private readonly uint _word0; + private readonly uint _word1; + private readonly uint _word2; + private readonly uint _word3; + private readonly uint _word4; + private readonly uint _word5; + private readonly uint _word6; + private readonly uint _word7; + + public SlotKey(DescriptorBindingKind kind, ulong view, ImageLayout layout, ReadOnlySpan words) + { + if (words.Length > MaxWords) + { + throw SubmissionScheduler.Fatal($"A bindless image descriptor has too many key words: {words.Length} > {MaxWords}."); + } + + Kind = kind; + View = view; + Layout = layout; + _wordCount = (uint)words.Length; + _word0 = words.Length > 0 ? words[0] : 0; + _word1 = words.Length > 1 ? words[1] : 0; + _word2 = words.Length > 2 ? words[2] : 0; + _word3 = words.Length > 3 ? words[3] : 0; + _word4 = words.Length > 4 ? words[4] : 0; + _word5 = words.Length > 5 ? words[5] : 0; + _word6 = words.Length > 6 ? words[6] : 0; + _word7 = words.Length > 7 ? words[7] : 0; + } + + public bool Equals(SlotKey other) => Kind == other.Kind && View == other.View && + Layout == other.Layout && + _wordCount == other._wordCount && + _word0 == other._word0 && _word1 == other._word1 && + _word2 == other._word2 && _word3 == other._word3 && + _word4 == other._word4 && _word5 == other._word5 && + _word6 == other._word6 && _word7 == other._word7; + + public override bool Equals(object? obj) => obj is SlotKey other && Equals(other); + + public override int GetHashCode() + { + var hash = new HashCode(); + hash.Add(Kind); + hash.Add(View); + hash.Add(Layout); + hash.Add(_wordCount); + hash.Add(_word0); + hash.Add(_word1); + hash.Add(_word2); + hash.Add(_word3); + hash.Add(_word4); + hash.Add(_word5); + hash.Add(_word6); + hash.Add(_word7); + return hash.ToHashCode(); + } + } + + private readonly GpuDeviceInfo _device; + private readonly SubmissionScheduler _scheduler; + private readonly DescriptorSet _set; + private readonly Dictionary _slots = new(); + private readonly Dictionary> _slotsByView = new(); + private readonly List[] _free = [[], [], []]; + private readonly Dictionary _samplerSlots = new(); + private readonly uint[] _next = new uint[BindingCount]; + private readonly uint[] _capacity = new uint[BindingCount]; + private readonly DescriptorSetLayout _layout; + + // The device's descriptor limits that bound the persistent heap. Samplers have their + // own per-stage and per-set limits and count against the pool-wide total too. + public readonly record struct Limits( + uint MaxPerStageSampledImages, + uint MaxPerStageStorageImages, + uint MaxPerStageSamplers, + uint MaxPerStageUpdateAfterBindSampledImages, + uint MaxPerStageUpdateAfterBindStorageImages, + uint MaxPerStageUpdateAfterBindSamplers, + uint MaxUpdateAfterBindSampledImages, + uint MaxUpdateAfterBindStorageImages, + uint MaxUpdateAfterBindSamplers, + uint MaxUpdateAfterBindDescriptors); + + // Sampled images, storage images and samplers that fit the limits. The samplers take + // their share of the pool-wide total first; the images split what remains. + public static (uint Sampled, uint Storage, uint Samplers) Capacities(in Limits limits) + { + var samplers = Math.Min(Math.Min(Math.Min(limits.MaxPerStageSamplers, limits.MaxPerStageUpdateAfterBindSamplers), + limits.MaxUpdateAfterBindSamplers), SamplerCapacity); + samplers = Math.Min(samplers, limits.MaxUpdateAfterBindDescriptors); + var sampledLimit = Math.Min( + Math.Min(Math.Min(limits.MaxPerStageSampledImages, limits.MaxPerStageUpdateAfterBindSampledImages), limits.MaxUpdateAfterBindSampledImages), + StableCapacityPerBinding); + var storageLimit = Math.Min( + Math.Min(Math.Min(limits.MaxPerStageStorageImages, limits.MaxPerStageUpdateAfterBindStorageImages), limits.MaxUpdateAfterBindStorageImages), + StableCapacityPerBinding); + var totalLimit = Math.Min((ulong)limits.MaxUpdateAfterBindDescriptors - samplers, (ulong)sampledLimit + storageLimit); + var sampledCapacity = Math.Min((ulong)sampledLimit, totalLimit / 2); + var storageCapacity = Math.Min((ulong)storageLimit, totalLimit - sampledCapacity); + if (sampledCapacity == 0 || storageCapacity == 0 || samplers == 0) + { + throw SubmissionScheduler.Fatal( + $"The device has no usable persistent image heap capacity: sampled={sampledLimit} storage={storageLimit} " + + $"samplers={samplers} total={limits.MaxUpdateAfterBindDescriptors}."); + } + + // If one type has a smaller limit, give the unused half to the other type. + var unassigned = totalLimit - sampledCapacity - storageCapacity; + if (unassigned != 0) + { + var sampledRoom = (ulong)sampledLimit - sampledCapacity; + var sampledExtra = Math.Min(unassigned, sampledRoom); + sampledCapacity += sampledExtra; + storageCapacity += Math.Min(unassigned - sampledExtra, (ulong)storageLimit - storageCapacity); + } + + return ((uint)sampledCapacity, (uint)storageCapacity, samplers); + } + + public BindlessImageHeap(GpuDeviceInfo device, SubmissionScheduler scheduler, in Limits limits) + { + _scheduler = scheduler; + var (sampledCapacity, storageCapacity, samplerCapacity) = Capacities(limits); + _capacity[0] = sampledCapacity; + _capacity[1] = storageCapacity; + _capacity[SamplerBinding] = samplerCapacity; + _device = device; + Console.Error.WriteLine( + $"[LOADER][INFO] Vulkan bindless heap sampled={sampledCapacity} storage={storageCapacity} samplers={samplerCapacity} " + + $"total={limits.MaxUpdateAfterBindDescriptors}"); + + var bindings = new DescriptorSetLayoutBinding[BindingCount]; + var flags = new DescriptorBindingFlags[bindings.Length]; + for (var index = 0u; index < bindings.Length; index++) + { + bindings[index] = new DescriptorSetLayoutBinding + { + Binding = index, + DescriptorType = index switch + { + 0 => DescriptorType.SampledImage, + 1 => DescriptorType.StorageImage, + _ => DescriptorType.Sampler, + }, + DescriptorCount = _capacity[index], + StageFlags = ShaderStageFlags.VertexBit | ShaderStageFlags.FragmentBit | ShaderStageFlags.ComputeBit | ShaderStageFlags.TessellationEvaluationBit, + }; + flags[index] = DescriptorBindingFlags.PartiallyBoundBit | + DescriptorBindingFlags.UpdateAfterBindBit | + DescriptorBindingFlags.UpdateUnusedWhilePendingBit; + } + + var bindingFlags = new DescriptorSetLayoutBindingFlagsCreateInfo + { + SType = StructureType.DescriptorSetLayoutBindingFlagsCreateInfo, + BindingCount = (uint)flags.Length, + }; + fixed (DescriptorSetLayoutBinding* bindingPointer = bindings) + fixed (DescriptorBindingFlags* flagPointer = flags) + { + bindingFlags.PBindingFlags = flagPointer; + var create = new DescriptorSetLayoutCreateInfo + { + SType = StructureType.DescriptorSetLayoutCreateInfo, + PNext = &bindingFlags, + Flags = DescriptorSetLayoutCreateFlags.UpdateAfterBindPoolBit, + BindingCount = (uint)bindings.Length, + PBindings = bindingPointer, + }; + Check(device.Vk.CreateDescriptorSetLayout(device.Device, &create, null, out _layout), "vkCreateDescriptorSetLayout(bindless)"); + } + + var poolSizes = new List(); + poolSizes.Add(new DescriptorPoolSize(DescriptorType.SampledImage, _capacity[0])); + poolSizes.Add(new DescriptorPoolSize(DescriptorType.StorageImage, _capacity[1])); + poolSizes.Add(new DescriptorPoolSize(DescriptorType.Sampler, _capacity[SamplerBinding])); + fixed (DescriptorPoolSize* poolPointer = poolSizes.ToArray()) + { + var poolInfo = new DescriptorPoolCreateInfo + { + SType = StructureType.DescriptorPoolCreateInfo, + Flags = DescriptorPoolCreateFlags.UpdateAfterBindBit, + MaxSets = 1, + PoolSizeCount = (uint)poolSizes.Count, + PPoolSizes = poolPointer, + }; + Check(device.Vk.CreateDescriptorPool(device.Device, &poolInfo, null, out var pool), "vkCreateDescriptorPool(bindless)"); + var setLayout = _layout; + var allocate = new DescriptorSetAllocateInfo + { + SType = StructureType.DescriptorSetAllocateInfo, + DescriptorPool = pool, + DescriptorSetCount = 1, + PSetLayouts = &setLayout, + }; + Check(device.Vk.AllocateDescriptorSets(device.Device, &allocate, out _set), "vkAllocateDescriptorSets(bindless)"); + _pool = pool; + } + + // Slot zero is the null descriptor. A draw maps an image to it only when the descriptor + // was rejected before residency, or a runtime lookup has not found it yet. + WriteNullDescriptor(0, 0); + WriteNullDescriptor(1, 0); + } + + // Samplers have no null descriptor: slot zero, which a missed runtime lookup reads, holds + // the host's default sampler. + public void SetDefaultSampler(Sampler sampler) => WriteSampler(0, sampler); + + public uint GetOrCreateSamplerSlot(Sampler sampler) + { + if (sampler.Handle == 0) + { + throw SubmissionScheduler.Fatal("A bindless sampler slot needs a sampler."); + } + + if (_samplerSlots.TryGetValue(sampler.Handle, out var existing)) + { + return existing; + } + + var slot = ++_next[SamplerBinding]; + if (slot >= _capacity[SamplerBinding]) + { + throw SubmissionScheduler.Fatal($"The bindless sampler heap is full: capacity={_capacity[SamplerBinding]}."); + } + + WriteSampler(slot, sampler); + _samplerSlots.Add(sampler.Handle, slot); + return slot; + } + + private void WriteSampler(uint slot, Sampler sampler) + { + var info = new DescriptorImageInfo { Sampler = sampler }; + var write = new WriteDescriptorSet + { + SType = StructureType.WriteDescriptorSet, + DstSet = _set, + DstBinding = SamplerBinding, + DstArrayElement = slot, + DescriptorCount = 1, + DescriptorType = DescriptorType.Sampler, + PImageInfo = &info, + }; + _device.Vk.UpdateDescriptorSets(_device.Device, 1, &write, 0, null); + } + + private DescriptorPool _pool; + + public DescriptorSetLayout Layout => _layout; + public DescriptorSet Set => _set; + + public uint GetOrCreateSlot(DescriptorBindingKind kind, ReadOnlySpan words, ImageView view, ImageLayout layout) + { + var index = DescriptorWriter.DescriptorType(kind) == DescriptorType.SampledImage ? 0u : 1u; + if (index >= _capacity.Length || view.Handle == 0) + { + throw SubmissionScheduler.Fatal($"The bindless image slot is invalid: kind={kind} view=0x{view.Handle:X}."); + } + + layout = index == 1 ? ImageLayout.General : layout; + var key = new SlotKey(kind, view.Handle, layout, words); + if (_slots.TryGetValue(key, out var existing)) + { + return existing; + } + + uint slot; + if (_free[index].Count != 0) + { + slot = _free[index][^1]; + _free[index].RemoveAt(_free[index].Count - 1); + } + else + { + slot = ++_next[index]; + if (slot >= _capacity[index]) + { + throw SubmissionScheduler.Fatal($"The bindless image heap is full: kind={kind} capacity={_capacity[index]}."); + } + } + + var info = new DescriptorImageInfo { ImageView = view, ImageLayout = layout }; + DescriptorImageInfo* infoPointer = &info; + { + var write = new WriteDescriptorSet + { + SType = StructureType.WriteDescriptorSet, + DstSet = _set, + DstBinding = index, + DstArrayElement = slot, + DescriptorCount = 1, + DescriptorType = DescriptorWriter.DescriptorType(kind), + PImageInfo = infoPointer, + }; + _device.Vk.UpdateDescriptorSets(_device.Device, 1, &write, 0, null); + } + _slots.Add(key, slot); + if (!_slotsByView.TryGetValue(view.Handle, out var viewSlots)) + { + viewSlots = []; + _slotsByView.Add(view.Handle, viewSlots); + } + viewSlots.Add(key); + return slot; + } + + public void InvalidateViews(IReadOnlyList views) + { + if (views.Count == 0) + { + return; + } + + var retired = new List<(uint Binding, uint Slot)>(); + foreach (var view in views) + { + if (!_slotsByView.Remove(view.Handle, out var viewSlots)) + { + continue; + } + + foreach (var key in viewSlots) + { + var binding = DescriptorWriter.DescriptorType(key.Kind) == DescriptorType.SampledImage ? 0u : 1u; + if (_slots.Remove(key, out var slot)) + { + retired.Add((binding, slot)); + } + } + } + + if (retired.Count != 0) + { + _scheduler.QueueCompletionAction(() => + { + foreach (var (binding, slot) in retired) + { + WriteNullDescriptor(binding, slot); + _free[binding].Add(slot); + } + }); + } + } + + private void WriteNullDescriptor(uint binding, uint slot) + { + var info = new DescriptorImageInfo + { + ImageLayout = binding == 1 ? ImageLayout.General : ImageLayout.Undefined, + }; + DescriptorImageInfo* infoPointer = &info; + var write = new WriteDescriptorSet + { + SType = StructureType.WriteDescriptorSet, + DstSet = _set, + DstBinding = binding, + DstArrayElement = slot, + DescriptorCount = 1, + DescriptorType = binding == 0 ? DescriptorType.SampledImage : DescriptorType.StorageImage, + PImageInfo = infoPointer, + }; + _device.Vk.UpdateDescriptorSets(_device.Device, 1, &write, 0, null); + } + + public void Dispose() + { + if (_pool.Handle != 0) _device.Vk.DestroyDescriptorPool(_device.Device, _pool, null); + if (_layout.Handle != 0) _device.Vk.DestroyDescriptorSetLayout(_device.Device, _layout, null); + _slots.Clear(); + _slotsByView.Clear(); + } + + private static void Check(Result result, string operation) + { + if (result != Result.Success) throw SubmissionScheduler.Fatal($"{operation} failed: result={result}."); + } +} diff --git a/src/SharpEmu.Libs/Gpu/Pipelines/BoundedFillDetector.cs b/src/SharpEmu.Libs/Gpu/Pipelines/BoundedFillDetector.cs index bab392a..1f688a4 100644 --- a/src/SharpEmu.Libs/Gpu/Pipelines/BoundedFillDetector.cs +++ b/src/SharpEmu.Libs/Gpu/Pipelines/BoundedFillDetector.cs @@ -160,6 +160,11 @@ public static class BoundedFillDetector return null; } + if (!HasPatternDataFlow(instructions)) + { + return null; + } + if (instructions[0].Destinations is not [{ Kind: Gen5OperandKind.VectorRegister } index] || instructions[0].Sources is not [{ Kind: Gen5OperandKind.ScalarRegister } group, { Kind: Gen5OperandKind.EncodedConstant, Value: Shift64 }, { Kind: Gen5OperandKind.VectorRegister, Value: ThreadIndexRegister }] || @@ -238,6 +243,144 @@ public static class BoundedFillDetector private static readonly (int Branch, int Target)[] PatternFillBranches = [(2, 53), (30, 49), (33, 44), (36, 39), (40, 43), (45, 48), (50, 53)]; + // The integer remainder idiom and its four mutually exclusive stores must + // agree on every operand, not just the opcodes. Names describe register + // roles and bind to the actual registers, so register allocation may vary. + // The empty side of a row denotes no explicit destination/source in the IR. + private static readonly (string Destinations, string Sources)[] PatternDataFlow = + [ + ("v:index", "s:group c:134 v:thread"), + ("", "s:count v:index"), + ("", ""), + ("v:quotient", "s:length"), + ("", "c:128 s:length"), + ("s:lengthMask", "exec c:128"), + ("v:quotient", "v:quotient"), + ("v:quotient", "l:4F800000 v:quotient"), + ("v:reciprocal", "v:quotient"), + ("v:quotient", "s:length v:reciprocal c:128"), + ("", "c:128 v:temporary"), + ("v:temporary", "c:128 v:quotient"), + ("v:quotient", "v:temporary v:quotient"), + ("v:temporary", "v:quotient v:reciprocal s:resource"), + ("v:quotient", "v:reciprocal v:temporary"), + ("v:temporary", "v:reciprocal v:temporary"), + ("v:quotient", "v:temporary v:quotient"), + ("v:quotient", "v:quotient v:index s:resource"), + ("v:temporary", "s:length v:quotient s:resource"), + ("v:reciprocal", "v:index v:temporary"), + ("", "s:length v:reciprocal"), + ("s:predicate", "v:index v:temporary"), + ("vcc", "s:predicate vcc"), + ("v:quotient", "c:128 v:quotient"), + ("v:quotient", "c:193 v:quotient s:predicate"), + ("v:quotient", "c:193 v:quotient s:lengthMask"), + ("v:quotient", "s:length v:quotient s:resource"), + ("v:quotient", "v:index v:quotient"), + ("", "c:128 v:quotient"), + ("s:outerMask", "vcc"), + ("", ""), + ("", "c:129 v:quotient"), + ("s:innerMask", "vcc"), + ("", ""), + ("", "c:130 v:quotient"), + ("vcc", "vcc"), + ("", ""), + ("v:data", "s:pattern3"), + ("v:data", "v:index s:resource c:128"), + ("exec", "vcc exec"), + ("", ""), + ("v:data", "s:pattern2"), + ("v:data", "v:index s:resource c:128"), + ("exec", "vcc"), + ("exec", "s:innerMask exec"), + ("", ""), + ("v:data", "s:pattern1"), + ("v:data", "v:index s:resource c:128"), + ("exec", "s:innerMask"), + ("exec", "s:outerMask exec"), + ("", ""), + ("v:data", "s:pattern0"), + ("v:data", "v:index s:resource c:128"), + ("", ""), + ]; + + private static bool HasPatternDataFlow(Gen5ShaderInstruction[] instructions) + { + var roles = new Dictionary + { + ["vcc"] = Gen5Operand.Scalar(106), + ["exec"] = Gen5Operand.Scalar(126), + ["v:thread"] = Gen5Operand.Vector(ThreadIndexRegister), + }; + bool Matches(IReadOnlyList operands, string pattern) + { + var names = pattern.Split(' ', StringSplitOptions.RemoveEmptyEntries); + if (operands.Count != names.Length) return false; + for (var index = 0; index < names.Length; index++) + { + var name = names[index]; + var operand = operands[index]; + if (roles.TryGetValue(name, out var bound)) + { + if (operand != bound) return false; + } + else if (name.StartsWith("c:", StringComparison.Ordinal)) + { + if (operand.Kind != Gen5OperandKind.EncodedConstant || operand.Value != uint.Parse(name.AsSpan(2))) return false; + } + else if (name.StartsWith("l:", StringComparison.Ordinal)) + { + if (operand.Kind != Gen5OperandKind.LiteralConstant || + operand.Value != uint.Parse(name.AsSpan(2), System.Globalization.NumberStyles.HexNumber)) return false; + } + else + { + var kind = name[0] == 's' ? Gen5OperandKind.ScalarRegister : Gen5OperandKind.VectorRegister; + if (operand.Kind != kind) return false; + roles.Add(name, operand); + } + } + return true; + } + + for (var at = 0; at < instructions.Length; at++) + { + var instruction = instructions[at]; + if (!Matches(instruction.Destinations, PatternDataFlow[at].Destinations) || + !Matches(instruction.Sources, PatternDataFlow[at].Sources)) return false; + switch (instruction.Control) + { + case null: + break; + case Gen5Vop3Control control when control.AbsoluteMask == 0 && control.NegateMask == 0 && + control.OutputModifier == 0 && !control.Clamp && control.OperandSelect == 0 && + control.ScalarDestination == (at is 9 or 24 ? 106u : (uint?)null): + break; + case Gen5SdwaControl control when at == 21 && control.DestinationSelect == 6 && control.DestinationUnused == 0 && + control.Source0Select == 6 && control.Source1Select == 6 && !control.Source0SignExtend && !control.Source1SignExtend && + control.AbsoluteMask == 0 && control.NegateMask == 0 && control.OutputModifier == 0 && !control.Clamp && + control.ScalarDestination == roles["s:predicate"].Value: + break; + case Gen5BufferMemoryControl control when at is 38 or 42 or 47 or 52 && !control.PackedD16 && control.ComponentCount == 1: + break; // The descriptor and address controls are checked by DetectPattern. + default: + return false; + } + } + + // MAD_U64 writes a consecutive pair; none of the remainder temporaries + // may overwrite the global index used by every store. + var quotient = roles["v:quotient"].Value; + var temporary = roles["v:temporary"].Value; + var indexRegister = roles["v:index"].Value; + var reciprocal = roles["v:reciprocal"].Value; + return temporary == quotient + 1 && new[] { quotient, temporary, indexRegister, reciprocal }.Distinct().Count() == 4 && + roles["v:data"].Value != indexRegister && + roles["s:predicate"].Value < 104 && roles["s:outerMask"].Value < 104 && roles["s:innerMask"].Value < 104 && + Math.Abs((long)roles["s:outerMask"].Value - roles["s:innerMask"].Value) >= 2; + } + private static HashSet WrittenScalars(Gen5ShaderInstruction[] instructions, int count) { var written = new HashSet(); diff --git a/src/SharpEmu.Libs/Gpu/Pipelines/ConstantFillDetector.cs b/src/SharpEmu.Libs/Gpu/Pipelines/ConstantFillDetector.cs index bc1f494..93db261 100644 --- a/src/SharpEmu.Libs/Gpu/Pipelines/ConstantFillDetector.cs +++ b/src/SharpEmu.Libs/Gpu/Pipelines/ConstantFillDetector.cs @@ -17,12 +17,35 @@ namespace SharpEmu.Libs.Gpu.Pipelines; // clear; the metadata is not emulated and the stores alone would leave the image stale. public sealed record ConstantFill(uint GroupScalarRegister, uint DestinationScalarResource, uint SourceScalarResource); +public sealed record ImmediateConstantFill(uint GroupScalarRegister, uint DestinationScalarResource); + public static class ConstantFillDetector { private const uint Shift64 = 134; // inline constant 6 private const uint ZeroConstant = 128; private const uint NullScalar = 125; + public static ImmediateConstantFill? DetectImmediate(Gen5ShaderProgram program) + { + var instructions = program.Instructions.Where(instruction => instruction.Opcode != "SWaitcnt").ToArray(); + if (instructions.Length != 4 || instructions[0].Opcode != "VLshlAddU32" || + instructions[0].Destinations is not [{ Kind: Gen5OperandKind.VectorRegister } index] || + instructions[0].Sources is not [{ Kind: Gen5OperandKind.ScalarRegister } group, + { Kind: Gen5OperandKind.EncodedConstant, Value: Shift64 }, { Kind: Gen5OperandKind.VectorRegister, Value: 0 }] || + instructions[1].Opcode != "VMovB32" || + instructions[1].Destinations is not [{ Kind: Gen5OperandKind.VectorRegister } data] || + data.Value == index.Value || + instructions[1].Sources is not [{ Kind: Gen5OperandKind.LiteralConstant or Gen5OperandKind.EncodedConstant }] || + instructions[2].Opcode != "BufferStoreFormatX" || + instructions[2].Control is not Gen5BufferMemoryControl + { DwordCount: 1, OffsetBytes: 0, IndexEnabled: true, OffsetEnabled: false, Typed: false } control || + control.VectorAddress != index.Value || control.VectorData != data.Value || + instructions[2].Sources is not [_, _, { Kind: Gen5OperandKind.EncodedConstant, Value: ZeroConstant }] || + instructions[3].Opcode != "SEndpgm") + return null; + return new ImmediateConstantFill(group.Value, control.ScalarResource); + } + public static ConstantFill? Detect(Gen5ShaderProgram program) { var instructions = program.Instructions.Where(instruction => instruction.Opcode != "SWaitcnt").ToArray(); @@ -34,7 +57,7 @@ public static class ConstantFillDetector var index = instructions[0]; if (index.Opcode != "VLshlAddU32" || index.Destinations is not [{ Kind: Gen5OperandKind.VectorRegister } indexRegister] || index.Sources is not [{ Kind: Gen5OperandKind.ScalarRegister } group, { Kind: Gen5OperandKind.EncodedConstant, Value: Shift64 }, - { Kind: Gen5OperandKind.VectorRegister } thread] || thread.Value != indexRegister.Value) + { Kind: Gen5OperandKind.VectorRegister } thread] || thread.Value != 0 || thread.Value != indexRegister.Value) { return null; } diff --git a/src/SharpEmu.Libs/Gpu/Pipelines/DescriptorHeap.cs b/src/SharpEmu.Libs/Gpu/Pipelines/DescriptorHeap.cs index 4531bb7..3e277bc 100644 --- a/src/SharpEmu.Libs/Gpu/Pipelines/DescriptorHeap.cs +++ b/src/SharpEmu.Libs/Gpu/Pipelines/DescriptorHeap.cs @@ -39,13 +39,6 @@ public sealed unsafe class DescriptorHeap : IDisposable public const uint SetsPerPool = 1024; public const uint SetBatch = 32; - private static readonly DescriptorPoolSize[] PoolSizes = - [ - new(DescriptorType.StorageBuffer, 8192), - new(DescriptorType.SampledImage, 8192), - new(DescriptorType.StorageImage, 1024), - new(DescriptorType.Sampler, 1024), - ]; private sealed class SetBatchState { @@ -56,11 +49,15 @@ public sealed unsafe class DescriptorHeap : IDisposable private readonly GpuDeviceInfo _device; private readonly SubmissionScheduler _scheduler; - private static readonly DescriptorSetDemand PoolCapacity = new(8192, 8192, 1024, 1024); + // An ordinary pool. A set larger than this gets a pool sized for a few of its kind, so the + // common case does not pay for the largest shader's descriptors. + public static readonly DescriptorSetDemand PoolCapacity = new(8192, 8192, 1024, 1024); + private const uint LargeSetsPerPool = 4; - private readonly Queue<(DescriptorPool Pool, ulong Tick)> _pendingPools = new(); + private readonly Queue<(DescriptorPool Pool, DescriptorSetDemand Capacity, ulong Tick)> _pendingPools = new(); private readonly Dictionary _sets = new(); private DescriptorPool _currentPool; + private DescriptorSetDemand _currentCapacity = PoolCapacity; private DescriptorSetDemand _remaining = PoolCapacity; private uint _remainingSets = SetsPerPool; @@ -68,13 +65,31 @@ public sealed unsafe class DescriptorHeap : IDisposable { _device = device; _scheduler = scheduler; - _currentPool = CreatePool(); + _currentPool = CreatePool(PoolCapacity); } public int PendingPoolCount => _pendingPools.Count; public DescriptorPool CurrentPool => _currentPool; + public DescriptorSetDemand CurrentCapacity => _currentCapacity; + + // The capacity of a pool that holds at least one set of the demand. + public static DescriptorSetDemand CapacityFor(in DescriptorSetDemand demand) + { + if (demand.Fits(PoolCapacity)) + { + return PoolCapacity; + } + + var large = demand.Scale(LargeSetsPerPool); + return new DescriptorSetDemand( + Math.Max(PoolCapacity.StorageBuffers, large.StorageBuffers), + Math.Max(PoolCapacity.SampledImages, large.SampledImages), + Math.Max(PoolCapacity.StorageImages, large.StorageImages), + Math.Max(PoolCapacity.Samplers, large.Samplers)); + } + public DescriptorSet Commit(DescriptorSetLayout layout, in DescriptorSetDemand demand) { if (layout.Handle == 0) @@ -98,10 +113,11 @@ public sealed unsafe class DescriptorHeap : IDisposable return batch.Sets[--batch.Size]; } - _pendingPools.Enqueue((_currentPool, _scheduler.CurrentTick)); - var (oldest, tick) = _pendingPools.Peek(); + _pendingPools.Enqueue((_currentPool, _currentCapacity, _scheduler.CurrentTick)); + var (oldest, oldestCapacity, tick) = _pendingPools.Peek(); + var needed = CapacityFor(demand); _scheduler.Timeline.RefreshCompletedTick(); - if (_scheduler.Timeline.IsTickComplete(tick)) + if (_scheduler.Timeline.IsTickComplete(tick) && needed.Fits(oldestCapacity)) { _pendingPools.Dequeue(); _currentPool = oldest; @@ -111,12 +127,13 @@ public sealed unsafe class DescriptorHeap : IDisposable throw SubmissionScheduler.Fatal($"vkResetDescriptorPool failed: result={reset}."); } - _remaining = PoolCapacity; + _currentCapacity = oldestCapacity; + _remaining = oldestCapacity; _remainingSets = SetsPerPool; } else { - _currentPool = CreatePool(); + _currentPool = CreatePool(needed); } _sets.Clear(); @@ -188,15 +205,22 @@ public sealed unsafe class DescriptorHeap : IDisposable } } - private DescriptorPool CreatePool() + private DescriptorPool CreatePool(in DescriptorSetDemand capacity) { - fixed (DescriptorPoolSize* poolSizes = PoolSizes) + DescriptorPoolSize[] sizes = + [ + new(DescriptorType.StorageBuffer, capacity.StorageBuffers), + new(DescriptorType.SampledImage, capacity.SampledImages), + new(DescriptorType.StorageImage, capacity.StorageImages), + new(DescriptorType.Sampler, capacity.Samplers), + ]; + fixed (DescriptorPoolSize* poolSizes = sizes) { var create = new DescriptorPoolCreateInfo { SType = StructureType.DescriptorPoolCreateInfo, MaxSets = SetsPerPool, - PoolSizeCount = (uint)PoolSizes.Length, + PoolSizeCount = (uint)sizes.Length, PPoolSizes = poolSizes, }; var result = _device.Vk.CreateDescriptorPool(_device.Device, &create, null, out var pool); @@ -205,7 +229,8 @@ public sealed unsafe class DescriptorHeap : IDisposable throw SubmissionScheduler.Fatal($"vkCreateDescriptorPool failed: result={result}."); } - _remaining = PoolCapacity; + _currentCapacity = capacity; + _remaining = capacity; _remainingSets = SetsPerPool; return pool; } @@ -216,7 +241,7 @@ public sealed unsafe class DescriptorHeap : IDisposable _device.Vk.DestroyDescriptorPool(_device.Device, _currentPool, null); while (_pendingPools.Count != 0) { - var (pool, tick) = _pendingPools.Dequeue(); + var (pool, _, tick) = _pendingPools.Dequeue(); _scheduler.Timeline.Wait(tick); _device.Vk.DestroyDescriptorPool(_device.Device, pool, null); } diff --git a/src/SharpEmu.Libs/Gpu/Pipelines/DescriptorWriter.cs b/src/SharpEmu.Libs/Gpu/Pipelines/DescriptorWriter.cs index 7a2d378..656c0d5 100644 --- a/src/SharpEmu.Libs/Gpu/Pipelines/DescriptorWriter.cs +++ b/src/SharpEmu.Libs/Gpu/Pipelines/DescriptorWriter.cs @@ -28,7 +28,8 @@ public static class DescriptorWriter { DescriptorBindingKind.Samplers => Silk.NET.Vulkan.DescriptorType.Sampler, DescriptorBindingKind.Buffers or DescriptorBindingKind.GlobalDataShare or DescriptorBindingKind.DeviceAddressPageTable or - DescriptorBindingKind.FaultBuffer or DescriptorBindingKind.FlattenedResourceTable or DescriptorBindingKind.ShaderData => + DescriptorBindingKind.FaultBuffer or DescriptorBindingKind.FlattenedResourceTable or DescriptorBindingKind.ShaderData or + DescriptorBindingKind.RuntimeDescriptorTable or DescriptorBindingKind.RuntimeDescriptorMisses => Silk.NET.Vulkan.DescriptorType.StorageBuffer, _ => throw SubmissionScheduler.Fatal($"The descriptor binding kind is invalid: kind={kind}."), }; @@ -41,12 +42,17 @@ public static class DescriptorWriter ShaderStage.Vertex => ShaderStageFlags.VertexBit, ShaderStage.Pixel => ShaderStageFlags.FragmentBit, ShaderStage.Compute => ShaderStageFlags.ComputeBit, + ShaderStage.TessellationEvaluation => ShaderStageFlags.TessellationEvaluationBit, _ => throw SubmissionScheduler.Fatal($"The shader stage is unknown: stage={stage}."), }; public static PipelineStageFlags PipelineStageFlag(ShaderStageFlags stages) { var result = PipelineStageFlags.None; + if ((stages & ShaderStageFlags.TessellationControlBit) != 0) + result |= PipelineStageFlags.TessellationControlShaderBit; + if ((stages & ShaderStageFlags.TessellationEvaluationBit) != 0) + result |= PipelineStageFlags.TessellationEvaluationShaderBit; if ((stages & ShaderStageFlags.VertexBit) != 0) { result |= PipelineStageFlags.VertexShaderBit; diff --git a/src/SharpEmu.Libs/Gpu/Pipelines/IShaderPipelineHost.cs b/src/SharpEmu.Libs/Gpu/Pipelines/IShaderPipelineHost.cs index 2eff4e4..fb5d6ff 100644 --- a/src/SharpEmu.Libs/Gpu/Pipelines/IShaderPipelineHost.cs +++ b/src/SharpEmu.Libs/Gpu/Pipelines/IShaderPipelineHost.cs @@ -11,6 +11,7 @@ namespace SharpEmu.Libs.Gpu.Pipelines; // Everything a host pipeline creation reads for one graphics pipeline. public sealed class GraphicsPipelineDescription { + public TessellationPipelineStages? Tessellation { get; init; } public required PipelineRenderingState Rendering { get; init; } public required PipelineVertexInputState VertexInput { get; init; } public required VertexInputInfo VertexInfo { get; init; } @@ -22,6 +23,8 @@ public sealed class GraphicsPipelineDescription public required PipelineStaticParameters StaticParameters { get; init; } } +public sealed record TessellationPipelineStages(ShaderProgram Control, ShaderProgram Evaluation, uint InputControlPoints); + public sealed class ComputePipelineDescription { public required ComputeInputInfo Input { get; init; } @@ -41,11 +44,32 @@ internal interface IShaderPipelineHost // The device supports shaderSharedInt64Atomics, so LDS 64-bit atomics can be // emitted as real 64-bit atomics instead of a non-atomic 32-bit pair. bool SharedInt64AtomicsEnabled { get; } + + // The device converts f16<->f32 natively with the hardware's rounding and denormals + // (see VulkanFloat16Support). + bool ExactFloat16ConversionsEnabled => false; + + bool NonUniformImageIndexingEnabled => false; + + // The host writes buffer strides into shader data (BindingLayout.UsesRuntimeBufferStrides), + // so the stride leaves the permutation key. + bool RuntimeBufferStridesEnabled => false; + bool UsesBindlessImages => false; bool ShaderSignedZeroInfNanPreserveFloat32Supported => false; bool ExecGuardElisionEnabled => true; ShaderPrewarmList? ShaderPrewarm => null; bool PerVertexPixelInputsSupported => true; bool ClipDistanceEnabled => false; + bool PostDepthCoverageSupported => false; + bool NativeTwoSampleMixedSupported => false; + + // True when this device's GLSL UnpackHalf2x16 / PackHalf2x16 were measured bit-exact + // against the translator's own f16 conversion. False for every host that did not measure it. + bool NativeHalfConversionExact => false; + + // True only when the host measured that a storage-buffer read past its descriptor range + // returns zero on this device; the translator then stops bounds-checking every guest word. + bool ZeroOutOfBoundsBufferReads => false; RenderHostLimits Limits { get; } @@ -60,6 +84,16 @@ internal interface IShaderPipelineHost // Reads one guest dword only when no GPU work may still own the range. bool TryReadCleanGuestWord(ulong address, out uint word); + // The device can blend with a second fragment output (dualSrcBlend). + bool SupportsDualSourceBlend => false; + + bool TryReadCleanGuestWords(ulong address, Span words) + { + using var profile = ResourceMaterializationProfile.Measure(ResourceMaterializationProfile.Phase.CleanGuestRead); + return BitConverter.IsLittleEndian && TryReadResidentGuestBytes(address, + System.Runtime.InteropServices.MemoryMarshal.AsBytes(words), clean: true); + } + // Copies guest bytes the CPU already holds, without synchronizing. False when the GPU // may own the range (or, for a clean read, when a clean word read would be refused); // the resource cache then re-materializes instead of trusting a stale copy. diff --git a/src/SharpEmu.Libs/Gpu/Pipelines/PipelineCacheSignature.cs b/src/SharpEmu.Libs/Gpu/Pipelines/PipelineCacheSignature.cs index 1421bc5..a0a98fa 100644 --- a/src/SharpEmu.Libs/Gpu/Pipelines/PipelineCacheSignature.cs +++ b/src/SharpEmu.Libs/Gpu/Pipelines/PipelineCacheSignature.cs @@ -40,7 +40,10 @@ public static class PipelineCacheSignature } // The payload when the signature and the hash match; false for any other file. - public static bool TryUnwrap(string signature, ReadOnlySpan file, out byte[] payload) + public static bool TryUnwrap(string signature, ReadOnlySpan file, out byte[] payload) => + TryUnwrap(signature, file, out payload, checkHash: true); + + private static bool TryUnwrap(string signature, ReadOnlySpan file, out byte[] payload, bool checkHash) { payload = []; if (file.Length > int.MaxValue) @@ -81,7 +84,7 @@ public static class PipelineCacheSignature var expectedHash = BinaryPrimitives.ReadUInt64LittleEndian(file[hashOffset..]); var data = file[(hashOffset + sizeof(ulong))..]; - if (XxHash3.HashToUInt64(data) != expectedHash) + if (checkHash && XxHash3.HashToUInt64(data) != expectedHash) { return false; } @@ -89,4 +92,66 @@ public static class PipelineCacheSignature payload = data.ToArray(); return true; } + + // Driver caches of large titles pass 2 GiB, beyond a .NET array, so these stream + // the same file format from and to native memory in chunks. + private const int StreamChunk = 64 << 20; + + // Writes the signature line, the payload hash and the payload. + public static unsafe void WriteTo(Stream stream, string signature, byte* payload, ulong length) + { + stream.Write(Encoding.ASCII.GetBytes(signature)); + var hash = new XxHash3(); + for (ulong offset = 0; offset < length; offset += StreamChunk) + hash.Append(new ReadOnlySpan(payload + offset, (int)Math.Min(StreamChunk, length - offset))); + Span hashBytes = stackalloc byte[sizeof(ulong)]; + BinaryPrimitives.WriteUInt64LittleEndian(hashBytes, hash.GetCurrentHashAsUInt64()); + stream.Write(hashBytes); + for (ulong offset = 0; offset < length; offset += StreamChunk) + stream.Write(new ReadOnlySpan(payload + offset, (int)Math.Min(StreamChunk, length - offset))); + } + + // Reads the payload into native memory (free it with NativeMemory.Free) when the + // signature (as TryUnwrap accepts it) and the hash match. + public static unsafe bool TryReadFrom(Stream stream, string signature, out byte* payload, out ulong length) + { + payload = null; + length = 0; + var line = new List(256); + int value; + while ((value = stream.ReadByte()) >= 0) + { + line.Add((byte)value); + if (value == '\n' || line.Count > 4096) break; + } + + if (line.Count == 0 || line[^1] != (byte)'\n' || + !TryUnwrap(signature, [.. line, .. new byte[sizeof(ulong)]], out _, checkHash: false)) + { + return false; + } + + Span hashBytes = stackalloc byte[sizeof(ulong)]; + stream.ReadExactly(hashBytes); + var expectedHash = BinaryPrimitives.ReadUInt64LittleEndian(hashBytes); + var remaining = (ulong)(stream.Length - stream.Position); + var data = (byte*)System.Runtime.InteropServices.NativeMemory.Alloc((nuint)Math.Max(remaining, 1)); + var hash = new XxHash3(); + for (ulong offset = 0; offset < remaining; offset += StreamChunk) + { + var chunk = new Span(data + offset, (int)Math.Min(StreamChunk, remaining - offset)); + stream.ReadExactly(chunk); + hash.Append(chunk); + } + + if (hash.GetCurrentHashAsUInt64() != expectedHash) + { + System.Runtime.InteropServices.NativeMemory.Free(data); + return false; + } + + payload = data; + length = remaining; + return true; + } } diff --git a/src/SharpEmu.Libs/Gpu/Pipelines/PipelineKeys.cs b/src/SharpEmu.Libs/Gpu/Pipelines/PipelineKeys.cs index f4eba84..705dbc2 100644 --- a/src/SharpEmu.Libs/Gpu/Pipelines/PipelineKeys.cs +++ b/src/SharpEmu.Libs/Gpu/Pipelines/PipelineKeys.cs @@ -21,8 +21,8 @@ public sealed class PipelineStaticParameters : IEquatable BinaryPrimitives.WriteUInt32LittleEndian(_bytes.AsSpan(offset), value); - private float GetFloat(int offset) => BinaryPrimitives.ReadSingleLittleEndian(_bytes.AsSpan(offset)); - - private void SetFloat(int offset, float value) => BinaryPrimitives.WriteSingleLittleEndian(_bytes.AsSpan(offset), value); - private StencilOperations GetStencil(int offset) => new( (StencilOp)GetUInt(offset), (StencilOp)GetUInt(offset + 4), @@ -95,8 +91,6 @@ public sealed class PipelineStaticParameters : IEquatable GetBool(SampleShadingOffset); set => SetBool(SampleShadingOffset, value); } public bool WithDepth { get => GetBool(WithDepthOffset); set => SetBool(WithDepthOffset, value); } public bool DepthBoundsTestEnable { get => GetBool(DepthBoundsTestOffset); set => SetBool(DepthBoundsTestOffset, value); } - public float DepthMinBounds { get => GetFloat(DepthMinBoundsOffset); set => SetFloat(DepthMinBoundsOffset, value); } - public float DepthMaxBounds { get => GetFloat(DepthMaxBoundsOffset); set => SetFloat(DepthMaxBoundsOffset, value); } public bool StencilTestEnable { get => GetBool(StencilTestOffset); set => SetBool(StencilTestOffset, value); } public StencilOperations StencilFront { get => GetStencil(StencilFrontOffset); set => SetStencil(StencilFrontOffset, in value); } public StencilOperations StencilBack { get => GetStencil(StencilBackOffset); set => SetStencil(StencilBackOffset, in value); } @@ -138,17 +132,21 @@ public sealed class PipelineStaticParameters : IEquatable { public Format[] ColorFormats { get; } = new Format[PipelineStaticParameters.ColorAttachmentCount]; + public uint[] ColorSamples { get; } = new uint[PipelineStaticParameters.ColorAttachmentCount]; public Format DepthFormat { get; set; } = Format.Undefined; public Format StencilFormat { get; set; } = Format.Undefined; + public uint DepthSamples { get; set; } + public uint[] SampleLocationWords { get; } = new uint[16]; public uint ColorCount { get; set; } public bool Equals(PipelineRenderingState? other) => other is not null && ColorCount == other.ColorCount && DepthFormat == other.DepthFormat && StencilFormat == other.StencilFormat && - ColorFormats.AsSpan().SequenceEqual(other.ColorFormats); + DepthSamples == other.DepthSamples && ColorFormats.AsSpan().SequenceEqual(other.ColorFormats) && + ColorSamples.AsSpan().SequenceEqual(other.ColorSamples) && SampleLocationWords.AsSpan().SequenceEqual(other.SampleLocationWords); public override bool Equals(object? obj) => Equals(obj as PipelineRenderingState); @@ -159,10 +157,13 @@ public sealed class PipelineRenderingState : IEquatable for (var index = 0; index < ColorCount; index++) { hash.Add(ColorFormats[index]); + hash.Add(ColorSamples[index]); } hash.Add(DepthFormat); hash.Add(StencilFormat); + hash.Add(DepthSamples); + foreach (var word in SampleLocationWords) hash.Add(word); return hash.ToHashCode(); } } @@ -209,16 +210,17 @@ public sealed class GraphicsPipelineKey : IEquatable public required PipelineRenderingState Rendering { get; init; } public ulong VertexProgramId { get; init; } public ulong PixelProgramId { get; init; } + public TessellationPipelineStages? Tessellation { get; init; } public required PipelineVertexInputState VertexInput { get; init; } public required PipelineStaticParameters StaticParameters { get; init; } public bool Equals(GraphicsPipelineKey? other) => other is not null && Rendering.Equals(other.Rendering) && VertexProgramId == other.VertexProgramId && PixelProgramId == other.PixelProgramId && - VertexInput.Equals(other.VertexInput) && StaticParameters.Equals(other.StaticParameters); + Tessellation == other.Tessellation && VertexInput.Equals(other.VertexInput) && StaticParameters.Equals(other.StaticParameters); public override bool Equals(object? obj) => Equals(obj as GraphicsPipelineKey); - public override int GetHashCode() => HashCode.Combine(Rendering, VertexProgramId, PixelProgramId, VertexInput, StaticParameters); + public override int GetHashCode() => HashCode.Combine(Rendering, VertexProgramId, PixelProgramId, VertexInput, StaticParameters, Tessellation); } public readonly record struct ComputePipelineKey(ulong ComputeProgramId); diff --git a/src/SharpEmu.Libs/Gpu/Pipelines/PixelStageInputResolver.cs b/src/SharpEmu.Libs/Gpu/Pipelines/PixelStageInputResolver.cs index 34e2d43..d0ecc38 100644 --- a/src/SharpEmu.Libs/Gpu/Pipelines/PixelStageInputResolver.cs +++ b/src/SharpEmu.Libs/Gpu/Pipelines/PixelStageInputResolver.cs @@ -63,9 +63,20 @@ public static class PixelStageInputResolver ShaderInterfaceRegisters shaderInterface, ReadOnlySpan targetOutputModes, ReadOnlySpan targetExportMapping, - uint inputCount) + uint inputCount, + uint maskExportSamples = 1, + uint rasterizationSamples = 1, + uint? pixelShaderIterationSamples = null, + uint shaderSampleExclusionMask = 0, + IReadOnlyList<(float X, float Y)>? customSampleOffsets = null, + bool forceShaderDepthOrder = false) { var activeInputs = shaderInterface.PixelInputEnable & shaderInterface.PixelInputAddress; + if (pixelShaderIterationSamples is uint iterations && + (iterations == 0 || rasterizationSamples == 0 || iterations > rasterizationSamples || + (iterations != 1 && iterations != rasterizationSamples))) + throw SubmissionScheduler.Fatal($"Unsupported pixel-shader iteration counts: iterations={iterations} raster={rasterizationSamples}."); + var customMask = 0u; var semanticCount = Math.Min(Math.Min(shader.InputSemanticsCount, inputCount), (uint)PixelInputInfo.InterpolatorCount); for (var index = 0u; index < semanticCount; index++) @@ -100,6 +111,16 @@ public static class PixelStageInputResolver } var control = shaderInterface.DepthShaderControl; + var earlyDepth = control.DepthExportOrder == 1 && !control.KillEnable && !control.DepthExportEnable && + !control.MaskExportEnable && !control.DepthBeforeShader && !control.DualExportEnable && !control.ExecuteOnNoop && control.RemainingBits == 0; + // Early-then-late order expresses a preference. Depth exports and coverage + // changes require late processing unless the guest forces shader ordering. + var requiresLateDepth = !forceShaderDepthOrder && control.DepthExportOrder == 1 && + (control.DepthExportEnable || control.MaskExportEnable || control.KillEnable) && !control.DepthBeforeShader && + !control.DualExportEnable && !control.ExecuteOnNoop && control.RemainingBits == 0; + if ((shaderSampleExclusionMask & 0xFFFFu) != 0 && control.DepthExportOrder == 1 && + !earlyDepth && !requiresLateDepth) + throw SubmissionScheduler.Fatal("Sample exclusion requires supported early depth/stencil processing."); return new PixelInputInfo { InputCount = inputCount, @@ -110,6 +131,7 @@ public static class PixelStageInputResolver : PixelInputInfo.NoPerspectiveCenterRegister, InterpolatorSettings = interpolators, TargetOutputModes = outputModes, + TargetExportFormats = (byte[])shaderInterface.TargetOutputModes.Clone(), TargetExportMappings = mappings, ScratchDwords = shader.ScratchDwords, PositionX = (activeInputs & InputPositionX) != 0, @@ -117,12 +139,20 @@ public static class PixelStageInputResolver PositionZ = (activeInputs & InputPositionZ) != 0, PositionW = (activeInputs & InputPositionW) != 0, FrontFace = (activeInputs & InputFrontFace) != 0, - SampleShading = (activeInputs & (InputPerspectiveSample | InputLinearSample)) != 0, + CustomSampleOffsets = (activeInputs & (InputPerspectiveSample | InputLinearSample)) != 0 + ? customSampleOffsets ?? [] : [], + SampleShading = pixelShaderIterationSamples is uint iterationCount + ? iterationCount > 1 : (activeInputs & (InputPerspectiveSample | InputLinearSample)) != 0, + InterpolationSample = pixelShaderIterationSamples == 1 && rasterizationSamples > 1 && + (activeInputs & (InputPerspectiveSample | InputLinearSample)) != 0 ? 0u : null, NoPerspective = (activeInputs & InputLinearCenter) != 0, KillEnable = control.KillEnable, DepthExportEnable = control.DepthExportEnable, SampleMaskExportEnable = control.MaskExportEnable, - EarlyDepth = control.DepthExportOrder == 1 && !control.KillEnable && !control.DepthExportEnable && !control.MaskExportEnable, + MaskExportSamples = maskExportSamples, + RasterizationSamples = rasterizationSamples, + EarlyDepth = earlyDepth, + ShaderSampleExclusionMask = earlyDepth ? shaderSampleExclusionMask & 0xFFFFu : 0u, ExecuteOnNoop = control.ExecuteOnNoop, }; } diff --git a/src/SharpEmu.Libs/Gpu/Pipelines/ShaderPermutationDump.cs b/src/SharpEmu.Libs/Gpu/Pipelines/ShaderPermutationDump.cs index a74faff..c93faa4 100644 --- a/src/SharpEmu.Libs/Gpu/Pipelines/ShaderPermutationDump.cs +++ b/src/SharpEmu.Libs/Gpu/Pipelines/ShaderPermutationDump.cs @@ -67,6 +67,8 @@ internal static class ShaderPermutationDump request.Bindings.PushDataStartDword, request.Bindings.ShaderDataDwordCount, request.Bindings.UsesDispatchThreadLimits, + request.Bindings.UsesBindlessImages, + request.Bindings.UsesRuntimeBufferStrides, request.Bindings.DispatchThreadLimitsDword, request.LocalSizeX, request.LocalSizeY, @@ -76,6 +78,12 @@ internal static class ShaderPermutationDump request.ThreadCountZ, request.WaveSize, request.ScratchDwords, + request.UserDataBase, + request.EnableExecGuardElision, + request.SupportsSharedInt64Atomics, + request.SupportsExactFloat16Conversions, + request.SupportsNonUniformImageIndexing, + request.ComputeSystemRegisters, InstructionCount = request.Program.Instructions.Count, }; File.WriteAllText($"{basePath}.json", JsonSerializer.Serialize(evidence, JsonOptions)); diff --git a/src/SharpEmu.Libs/Gpu/Pipelines/ShaderPipelineCache.Tessellation.cs b/src/SharpEmu.Libs/Gpu/Pipelines/ShaderPipelineCache.Tessellation.cs new file mode 100644 index 0000000..665bee8 --- /dev/null +++ b/src/SharpEmu.Libs/Gpu/Pipelines/ShaderPipelineCache.Tessellation.cs @@ -0,0 +1,76 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.Libs.Gpu.GpuCommands.Registers; +using SharpEmu.Libs.Gpu.Rendering; +using SharpEmu.Libs.Gpu.Scheduling; +using SharpEmu.ShaderCompiler; +using SharpEmu.ShaderCompiler.Resources; + +namespace SharpEmu.Libs.Gpu.Pipelines; + +internal sealed partial class ShaderPipelineCache +{ + internal static uint[] MergedHullUserData(UserScalarRegisters registers, uint declaredCount, ulong backAddress) + { + var front = UserData(registers, declaredCount, false, 0, "merged-hull"); + // Carry the architectural s0:s1 pair independently of s8 and above. + // Lowering copies these appended inputs before executing guest code. + var data = new uint[front.Length + 2]; + front.CopyTo(data, 0); + data[front.Length] = (uint)backAddress; + data[front.Length + 1] = (uint)(backAddress >> 32); + return data; + } + + internal static bool IsMergedTessellationMask(uint stages) + { + // LS_EN=1, HS_EN=1, ES_EN=1, DYNAMIC_HS=1, PRIMGEN_EN=1, + // PRIMGEN_PASSTHRU_EN=1, without a geometry or legacy vertex stage. + const uint mask = 1u | (1u << 2) | (1u << 3) | (1u << 8) | (1u << 13) | (1u << 25); + return (stages & ~((1u << 22) | (1u << 23))) == mask; + } + + private TessellationDrawPrograms PrepareTessellationHull(VertexStageRegisters vertex, + ShaderInterfaceRegisters shaderInterface, Gen5TessellationInfo tessellation) + { + var address = vertex.LocalAddress != 0 ? vertex.LocalAddress : vertex.HullAddress; + var registered = _registry.Require(address, "merged-hull"); + if (!registered.IsFused) + throw SubmissionScheduler.Fatal("The tessellation path requires a registered merged local/hull program."); + var count = vertex.HullResource2.UserScalarCount; + // A merged front may consume more user data than the back's RSRC2. + // Reading its own declared count preserves the guest's AGC header. + if (!Gen5ShaderTranslator.TryGetFusedProgramParts(_context, address, out var frontHeader, out _, out _) || + !_context.TryReadUInt64(frontHeader + 0x20, out var shRegisters) || + !_context.TryReadUInt32(frontHeader + 0x5C, out var shCount)) + throw SubmissionScheduler.Fatal("The merged hull front has an unreadable register header."); + for (uint index = 0; index < (shCount & 0xFF); index++) + { + if (!_context.TryReadUInt32(shRegisters + index * 8, out var offset) || + !_context.TryReadUInt32(shRegisters + index * 8 + 4, out var value)) + throw SubmissionScheduler.Fatal("The merged hull front has an unreadable register entry."); + if (offset == ShaderRegisterOffset.SpiShaderPgmRsrc2Hs) + count = Math.Max(count, HullResource2.Decode(value).UserScalarCount); + } + var source = PrepareSource(address, ShaderStage.Compute, "merged-hull", vertex.HullUserScalars, + count, false, VertexUserDataBase); + source = source with { UserData = MergedHullUserData(vertex.HullUserScalars, count, vertex.HullUserDataAddress) }; + var decoded = _programs.Decode(source); + if (decoded.FusedContinuationPc is not { } entry || + !Gen5TessellationHullInfo.TryDecode(shaderInterface.LocalHullConfiguration, entry + 8, out var hull, out var error)) + throw SubmissionScheduler.Fatal("The merged hull has an invalid continuation or workgroup configuration."); + var threads = (hull.PatchesPerGroup * Math.Max(hull.InputControlPoints, hull.OutputControlPoints) + 63) & ~63u; + var input = new ComputeInputInfo + { + ThreadsX = threads, ThreadsY = 1, ThreadsZ = 1, WaveSize = 64, ThreadIdCount = 1, + LocalDataShareDwords = (uint)vertex.HullResource2.LocalDataShareSize * 128, + ScratchDwords = registered.ScratchDwords, NeedsLocalDataShareBarriers = true, + }; + var cursor = 0u; + var handle = _programs.GetOrCompile(source, new StageCompileOptions + { ComputeInfo = input, TessellationHull = hull }, ref cursor, out var stage); + input.Stage = stage; + return new(tessellation, hull, new ComputeProgram { Program = handle, Input = input }); + } +} diff --git a/src/SharpEmu.Libs/Gpu/Pipelines/ShaderPipelineCache.cs b/src/SharpEmu.Libs/Gpu/Pipelines/ShaderPipelineCache.cs index 96a8f8d..57d569d 100644 --- a/src/SharpEmu.Libs/Gpu/Pipelines/ShaderPipelineCache.cs +++ b/src/SharpEmu.Libs/Gpu/Pipelines/ShaderPipelineCache.cs @@ -40,6 +40,7 @@ internal sealed partial class ShaderPipelineCache : IShaderPipelineProvider public ShaderProgramCache Programs => _programs; + public int GraphicsPipelineCount => _graphicsPipelines.Count; public int ComputePipelineCount => _computePipelines.Count; @@ -80,10 +81,18 @@ internal sealed partial class ShaderPipelineCache : IShaderPipelineProvider bool depthBound) { using var profileScope = RenderPhaseProfile.MeasureDetail(RenderPhaseProfile.Phase.ProgramPreparation); + var tessellated = IsMergedTessellationMask(context.ShaderStages); + Gen5TessellationInfo? tessellation = null; + if (tessellated) + { + if (!Gen5TessellationInfo.TryDecode(shaderInterface.TessellationFactorParameter, out var configuration, out var error)) + throw SubmissionScheduler.Fatal(error); + tessellation = configuration; + } var vertexSource = PrepareSource( - vertex.ExportAddress, ShaderStage.Vertex, "vertex", vertex.GeometryUserScalars, vertex.GeometryResource2.UserScalarCount, + vertex.ExportAddress, tessellated ? ShaderStage.TessellationEvaluation : ShaderStage.Vertex, "vertex", vertex.GeometryUserScalars, vertex.GeometryResource2.UserScalarCount, probeWrittenRegisters: true, VertexUserDataBase); - var vertexInfo = PrepareVertexInput(vertexSource, shaderInterface, context); + var vertexInfo = tessellated ? new VertexInputInfo { PositionExportControl = shaderInterface.VertexOutputControl } : PrepareVertexInput(vertexSource, shaderInterface, context); ShaderSource? pixelSource = null; PixelInputInfo? pixelInfo = null; Gen5PixelOutputBinding[] pixelOutputs = []; @@ -112,7 +121,23 @@ internal sealed partial class ShaderPipelineCache : IShaderPipelineProvider pixelOutputs = ResolveBoundTargets(context, targetExportMapping, depthBound ? pixelProgram.PixelColorExportMasks : null, out var outputModes, out var outputMappings); - pixelInfo = PixelStageInputResolver.Resolve(_context, pixelSource.Registered, shaderInterface, outputModes, outputMappings, inputCount); + var customSampleOffsets = new List<(float X, float Y)>(); + var rasterizationSamples = 1u << context.AntialiasingConfig.SampleCountLog2; + var pixelIterations = (context.ScanModeControl1 & (1u << 16)) != 0 + ? 1u << context.EnhancedQualityAntialiasing.PixelShaderIterationSamples : 1u; + if (_host.NativeTwoSampleMixedSupported && rasterizationSamples == 2 && + (shaderInterface.PixelInputEnable & shaderInterface.PixelInputAddress & 0x11u) != 0) + for (uint pixelLocation = 0; pixelLocation < 4; pixelLocation++) + for (uint sample = 0; sample < (pixelIterations == 1 ? 1 : rasterizationSamples); sample++) + { + var position = context.SampleLocations.Position(pixelLocation, sample); + customSampleOffsets.Add((position.X - .5f, position.Y - .5f)); + } + pixelInfo = PixelStageInputResolver.Resolve(_context, pixelSource.Registered, shaderInterface, + outputModes, outputMappings, inputCount, + 1u << context.EnhancedQualityAntialiasing.MaskExportSamples, + rasterizationSamples, pixelIterations, context.ShaderSampleExclusionMask, customSampleOffsets, + context.DepthRenderOverride.ForceShaderDepthOrder); // SPI_PS_INPUT_CNTL can map an input to any parameter export, beyond the input count; // the vertex program must declare every location the pixel program reads. attributeCount = Math.Max(attributeCount, ReadVertexOutputCount(pixelProgram, pixelInfo)); @@ -144,7 +169,7 @@ internal sealed partial class ShaderPipelineCache : IShaderPipelineProvider if (!TryPrepareProgram( vertexSource, - new StageCompileOptions { VertexInfo = vertexInfo, RequiredVertexOutputCount = (int)attributeCount }, + new StageCompileOptions { VertexInfo = vertexInfo, RequiredVertexOutputCount = (int)attributeCount, Tessellation = tessellation }, ref pushDataCursor, out vertexProgram, out vertexStage)) @@ -156,10 +181,25 @@ internal sealed partial class ShaderPipelineCache : IShaderPipelineProvider { pixelInfo.Stage = pixelStage; } + TessellationDrawPrograms? tessellationPrograms = null; + if (tessellated) + { + lock (_gate) + { + tessellationPrograms = PrepareTessellationHull(vertex, shaderInterface, tessellation!.Value); + var bridge = _programs.GetTessellationBridge(vertexStage.Program!.Bindings!, tessellation.Value.Domain); + vertexInfo = new VertexInputInfo + { + PositionExportControl = vertexInfo.PositionExportControl, Stage = vertexStage, + Tessellation = new(bridge.Control, vertexProgram, tessellationPrograms.HullConfiguration.InputControlPoints), + }; + vertexProgram = bridge.Vertex; + } + } SolidColorClear? solidClear = null; var disableBlending = false; - if (pixelInfo is not null) + if (pixelInfo is not null && !tessellated) { var vertexProgramWords = _programs.Decode(vertexSource); var pixelProgramWords = _programs.Decode(pixelSource!); @@ -177,6 +217,7 @@ internal sealed partial class ShaderPipelineCache : IShaderPipelineProvider return new GraphicsPrograms { Vertex = vertexProgram, + Tessellation = tessellationPrograms, Pixel = pixelProgramHandle, VertexInput = vertexInfo, PixelInput = pixelInfo ?? new PixelInputInfo(), @@ -201,8 +242,8 @@ internal sealed partial class ShaderPipelineCache : IShaderPipelineProvider viewport.YScale, viewport.XOffset, viewport.YOffset, - Math.Min(limits.MaxViewportWidth, MaxViewportDimension) * 0.5f, - Math.Min(limits.MaxViewportHeight, MaxViewportDimension) * 0.5f); + Images.RenderScalePolicy.ClipSpaceReferenceExtent(Math.Min(limits.MaxViewportWidth, MaxViewportDimension)) * 0.5f, + Images.RenderScalePolicy.ClipSpaceReferenceExtent(Math.Min(limits.MaxViewportHeight, MaxViewportDimension)) * 0.5f); } return VertexInputResolver.ResolveVertexInputs(_context, source.Registered, source.UserData, @@ -210,15 +251,22 @@ internal sealed partial class ShaderPipelineCache : IShaderPipelineProvider } // One past the highest parameter location the pixel program reads, resolved as its translator does. - private static uint ReadVertexOutputCount(Gen5ShaderProgram pixelProgram, PixelInputInfo info) - { - var attributes = pixelProgram.Instructions + // The attributes a pixel program interpolates, ascending; scanned once per decoded program + // instead of on every draw. + private static readonly System.Runtime.CompilerServices.ConditionalWeakTable _interpolatedAttributes = new(); + + private static uint[] InterpolatedAttributes(Gen5ShaderProgram program) => + _interpolatedAttributes.GetValue(program, static program => program.Instructions .Select(static instruction => instruction.Control) .OfType() .Select(static control => control.Attribute) .Distinct() .Order() - .ToArray(); + .ToArray()); + + private static uint ReadVertexOutputCount(Gen5ShaderProgram pixelProgram, PixelInputInfo info) + { + var attributes = InterpolatedAttributes(pixelProgram); if (attributes.Length == 0) { return 0; @@ -237,16 +285,8 @@ internal sealed partial class ShaderPipelineCache : IShaderPipelineProvider private static uint InterpolatedAttributeCount(Gen5ShaderProgram program) { - var maxAttribute = -1; - foreach (var instruction in program.Instructions) - { - if (instruction.Control is Gen5InterpolationControl interpolation) - { - maxAttribute = Math.Max(maxAttribute, (int)interpolation.Attribute); - } - } - - return (uint)(maxAttribute + 1); + var attributes = InterpolatedAttributes(program); + return attributes.Length == 0 ? 0u : attributes[^1] + 1; } // The bound colour slots in order; each output mode names the kind the pixel program exports. @@ -293,12 +333,43 @@ internal sealed partial class ShaderPipelineCache : IShaderPipelineProvider outputs.Add(new Gen5PixelOutputBinding(slot, location++, kind, mapping) { ExportTarget = exportTarget >= 0 ? (uint)exportTarget : ContextRegisters.ColorTargetCount + slot, + ExportFormat = exportTarget >= 0 + ? (Gen5PixelExportFormat)context.ShaderInterface.TargetOutputModes[exportTarget] + : Gen5PixelExportFormat.Float16, }); } + // Dual-source blending reads its second source from the export after target 0's, + // as the hardware pairs MRT0 and MRT1. It shares target 0's location as index 1. + if (_host.SupportsDualSourceBlend && UsesSecondBlendSource(context.BlendControls[0]) && + outputs.Count == 1 && outputs[0].GuestSlot == 0 && outputs[0].ExportTarget < ContextRegisters.ColorTargetCount) + { + var first = outputs[0]; + outputs.Add(new Gen5PixelOutputBinding(1, first.HostLocation, first.Kind, first.ComponentMapping) + { + ExportTarget = first.ExportTarget + 1, + ExportFormat = first.ExportTarget + 1 < ContextRegisters.ColorTargetCount + ? (Gen5PixelExportFormat)context.ShaderInterface.TargetOutputModes[first.ExportTarget + 1] + : Gen5PixelExportFormat.Float16, + Index = 1, + }); + // Key the compiled program by the second source as well. + outputModes[1] = DualSourceOutputMode; + } + return outputs.ToArray(); } + private const byte DualSourceOutputMode = 0xFF; + + // BLEND_SRC1_COLOR, BLEND_ONE_MINUS_SRC1_COLOR, BLEND_SRC1_ALPHA, BLEND_ONE_MINUS_SRC1_ALPHA. + internal static bool UsesSecondBlendSource(in BlendRegisters blend) + { + static bool Second(byte factor) => factor is >= 15 and <= 18; + return blend.Enable && (Second(blend.ColorSourceFactor) || Second(blend.ColorDestinationFactor) || + Second(blend.AlphaSourceFactor) || Second(blend.AlphaDestinationFactor)); + } + private static VertexPositionStream? FindPositionStream(VertexInputInfo info) { foreach (var attribute in info.Attributes) @@ -419,7 +490,7 @@ internal sealed partial class ShaderPipelineCache : IShaderPipelineProvider using var profileScope = RenderPhaseProfile.MeasureDetail(RenderPhaseProfile.Phase.PipelineCreation); var description = BuildGraphicsDescription( colors, in depth, vertexInput, pixelInput, context, in rendering, topology, primitiveRestartEnabled, disableBlending, - vertexProgram, pixelProgram, _host.NoAttachmentSampleCounts); + vertexProgram, pixelProgram, _host.NoAttachmentSampleCounts, _host.NativeTwoSampleMixedSupported); var key = KeyOf(description); lock (_gate) { @@ -456,7 +527,7 @@ internal sealed partial class ShaderPipelineCache : IShaderPipelineProvider bool disableBlending, ShaderProgram vertexProgram, ShaderProgram pixelProgram, - SampleCountFlags noAttachmentSampleCounts) + SampleCountFlags noAttachmentSampleCounts, bool nativeTwoSampleMixedSupported = false) { if (colors.Length > PipelineStaticParameters.ColorAttachmentCount) { @@ -489,6 +560,7 @@ internal sealed partial class ShaderPipelineCache : IShaderPipelineProvider } renderingState.ColorFormats[index] = format; + renderingState.ColorSamples[index] = color.Resolution.Samples; // A target the pixel program never exports keeps its contents, as on hardware; the // host output would otherwise write an undefined value (e.g. depth-only passes that // leave a color target bound and export only to the null target). @@ -512,6 +584,18 @@ internal sealed partial class ShaderPipelineCache : IShaderPipelineProvider { renderingState.DepthFormat = rendering.DepthFormat; renderingState.StencilFormat = rendering.StencilFormat; + renderingState.DepthSamples = depth.Target.Target.Samples; + } + + if (nativeTwoSampleMixedSupported && rendering.Samples == 2) + { + // Coverage masks are per pixel in the 2x2 sample-location grid. + // The native path currently represents unrestricted coverage only. + const uint activeSamples = 0x0003_0003; + if ((context.SampleCoverageMaskX0Y0X1Y0 & activeSamples) != activeSamples || + (context.SampleCoverageMaskX0Y1X1Y1 & activeSamples) != activeSamples) + throw SubmissionScheduler.Fatal("Two-sample rendering with restricted per-pixel coverage is not supported."); + context.SampleLocations.Locations.CopyTo(renderingState.SampleLocationWords, 0); } var samples = rendering.Samples; @@ -531,7 +615,9 @@ internal sealed partial class ShaderPipelineCache : IShaderPipelineProvider } var depthState = withDepth ? depth.Target.State : default; - var rectangleList = topology == PrimitiveTopology.PatchList; + // A patch list without tessellation stages carries legacy rectangles, which are never + // culled. Tessellated patches rasterize as ordinary triangles and keep the guest's culling. + var rectangleList = topology == PrimitiveTopology.PatchList && vertexInput.Tessellation is null; var mode = context.RasterMode; parameters.NegativeOneToOne = !context.Clip.DirectXClipSpace; parameters.DepthClipEnable = context.Clip.IsZClipEnabled; @@ -541,8 +627,6 @@ internal sealed partial class ShaderPipelineCache : IShaderPipelineProvider parameters.SampleShadingEnable = pixelActive && samples > 1 && pixelInput!.SampleShading; parameters.WithDepth = withDepth; parameters.DepthBoundsTestEnable = depthState.DepthBoundsTestEnabled; - parameters.DepthMinBounds = depthState.DepthMinBounds; - parameters.DepthMaxBounds = depthState.DepthMaxBounds; parameters.StencilTestEnable = depthState.StencilTestEnabled; parameters.StencilFront = withDepth ? depthState.FrontOperations : StencilOperations.Default; parameters.StencilBack = withDepth ? depthState.BackOperations : StencilOperations.Default; @@ -566,6 +650,7 @@ internal sealed partial class ShaderPipelineCache : IShaderPipelineProvider return new GraphicsPipelineDescription { + Tessellation = vertexInput.Tessellation, Rendering = renderingState, VertexInput = BuildVertexInputState(vertexInput), VertexInfo = vertexInput, @@ -583,6 +668,7 @@ internal sealed partial class ShaderPipelineCache : IShaderPipelineProvider Rendering = description.Rendering, VertexProgramId = description.VertexProgram.Id, PixelProgramId = description.PixelInfo is not null ? description.PixelProgram.Id : 0, + Tessellation = description.Tessellation, VertexInput = description.VertexInput, StaticParameters = description.StaticParameters, }; diff --git a/src/SharpEmu.Libs/Gpu/Pipelines/ShaderPrewarmList.cs b/src/SharpEmu.Libs/Gpu/Pipelines/ShaderPrewarmList.cs index 980bcb1..b1cdece 100644 --- a/src/SharpEmu.Libs/Gpu/Pipelines/ShaderPrewarmList.cs +++ b/src/SharpEmu.Libs/Gpu/Pipelines/ShaderPrewarmList.cs @@ -145,7 +145,9 @@ internal sealed class ShaderPrewarmList : IDisposable public const string ProgressFileName = "shader-prewarm.progress"; private const uint Magic = 0x57504553; - private const uint FormatVersion = 1; + // Resource specializations depend on the planner's semantics, not only on the + // serialized property layout. Bump this when those semantics change. + private const uint FormatVersion = 2; private const byte CodeKind = 1; private const byte ComputeKind = 2; private const int RecordHeaderBytes = sizeof(uint) + sizeof(ulong); diff --git a/src/SharpEmu.Libs/Gpu/Pipelines/ShaderProgramCache.cs b/src/SharpEmu.Libs/Gpu/Pipelines/ShaderProgramCache.cs index 611b9c8..4dbba01 100644 --- a/src/SharpEmu.Libs/Gpu/Pipelines/ShaderProgramCache.cs +++ b/src/SharpEmu.Libs/Gpu/Pipelines/ShaderProgramCache.cs @@ -23,6 +23,7 @@ public sealed record ShaderSource(RegisteredShader Registered, ulong Hash, uint[ { ShaderStage.Vertex => "vertex", ShaderStage.Pixel => "pixel", + ShaderStage.TessellationEvaluation => "domain", _ => "compute", }; } @@ -38,6 +39,8 @@ public sealed class StageCompileOptions public uint PixelInputAddress { get; init; } public ComputeInputInfo? ComputeInfo { get; init; } public Gen5ComputeSystemRegisters? ComputeSystemRegisters { get; init; } + public Gen5TessellationInfo? Tessellation { get; init; } + public Gen5TessellationHullInfo? TessellationHull { get; init; } } // The key of a program entry: what the emitter reads besides the resource specialization. @@ -139,6 +142,21 @@ internal sealed class ShaderProgramCache public int ProgramCount => _programs.Count; public IEnumerable Entries => _programs.Values; + private readonly Dictionary<(BindingLayout Layout, Gen5TessellationDomain Domain), (ShaderProgram Vertex, ShaderProgram Control)> _tessellationBridges = []; + + public (ShaderProgram Vertex, ShaderProgram Control) GetTessellationBridge(BindingLayout layout, Gen5TessellationDomain domain) + { + if (_tessellationBridges.TryGetValue((layout, domain), out var bridge)) return bridge; + var shaders = _compiler.CompileTessellationBridge(layout, domain); + ShaderProgram Create(IGuestCompiledShader shader, ShaderStage stage) + { + var id = ++_nextProgramId; + return new(id, _host.CreateShaderModule(shader, stage, 0, id)); + } + bridge = (Create(shaders.Vertex, ShaderStage.Vertex), Create(shaders.Control, ShaderStage.TessellationEvaluation)); + _tessellationBridges.Add((layout, domain), bridge); + return bridge; + } // The decoded instructions of a program, shared by every static variant of the same code. public Gen5ShaderProgram Decode(ShaderSource source) @@ -234,8 +252,9 @@ internal sealed class ShaderProgramCache { UserData = source.UserData, ShaderBase = source.Address, - ReadMemory = _readGuestWord, - ReadCleanMemory = _readCleanGuestWord, + ReadMemory = _host.TryReadGuestWord, + ReadCleanMemory = _host.TryReadCleanGuestWord, + ReadCleanWords = _host.TryReadCleanGuestWords, ReadResidentMemory = _prefetchResidentGuestBytes, ComputeState = source.Stage == ShaderStage.Compute && options.ComputeInfo is { } computeState ? new ComputeSelectorState(computeState.WaveSize, Math.Max(computeState.ThreadsX, 1), @@ -270,10 +289,21 @@ internal sealed class ShaderProgramCache if (!materialized) { var message = $"The shader resources could not be materialized: stage={source.Label} hash=0x{source.Hash:X16} shader=0x{source.Address:X16} reason={materializationFailure}."; + // Set on this thread by the materialization that just failed; the cache only delegates to it. + if (ResourceMaterializer.LastFailureDetail is { } detail) + message = message[..^1] + $" detail={detail}."; if (materializationFailure is ResourceMaterializationFailure.IncompatibleImageCandidates or ResourceMaterializationFailure.ImageCapacityExceeded) throw new ShaderProgramRejectedException(message); throw SubmissionScheduler.Fatal(message); } + + if (_host.RuntimeBufferStridesEnabled) + { + for (var index = 0; index < specialization.Buffers.Count; index++) + { + specialization.Buffers[index] = specialization.Buffers[index].WithoutRuntimeStride(); + } + } } using (RenderPhaseProfile.MeasureDetail(RenderPhaseProfile.Phase.ProgramPermutationLookup)) @@ -281,7 +311,7 @@ internal sealed class ShaderProgramCache foreach (var candidate in entry.Permutations) { var layout = candidate.Bindings; - if (layout.PushDataStartDword == PushData.StartFor(pushDataCursor, layout.ShaderDataDwordCount) && candidate.Specialization.Equals(specialization)) + if (layout.PushDataStartDword == (layout.UsesTessellationData ? PushData.NoStart : PushData.StartFor(pushDataCursor, layout.ShaderDataDwordCount)) && candidate.Specialization.Equals(specialization)) { stage = CreateStageResources(candidate.Program, snapshot, source, options); layout.AdvancePushData(ref pushDataCursor); @@ -325,6 +355,7 @@ internal sealed class ShaderProgramCache switch (stage) { case ShaderStage.Vertex: + case ShaderStage.TessellationEvaluation: StageStaticKey.Build(options.VertexInfo ?? throw new ArgumentException("The vertex lookup has no vertex input info."), options.RequiredVertexOutputCount, _staticState); break; case ShaderStage.Pixel: @@ -334,11 +365,26 @@ internal sealed class ShaderProgramCache StageStaticKey.Build(options.ComputeInfo ?? throw new ArgumentException("The compute lookup has no compute input info."), _staticState); break; } + if (options.Tessellation is { } tessellation) + { + _staticState.Add((uint)tessellation.Domain); + _staticState.Add((uint)tessellation.Spacing); + _staticState.Add(tessellation.PointMode ? 1u : 0); + _staticState.Add(tessellation.Clockwise ? 1u : 0); + } + if (options.TessellationHull is { } hull) + { + _staticState.Add(hull.InputControlPoints); _staticState.Add(hull.OutputControlPoints); + _staticState.Add(hull.PatchesPerGroup); _staticState.Add(hull.HullEntryPc); + } } private ProgramSourceEntry CreateEntry(ShaderSource source, StageCompileOptions options) { var program = Decode(source); + if (options.TessellationHull is not null) + program = Gen5TessellationLowering.PrepareMergedHull(program, + checked(source.UserDataBase + (uint)source.UserData.Length - 2)); var dumpPlanning = CompiledShaderDump.ShouldWrite(source.Address, source.Hash); if (dumpPlanning) ShaderPlanningDump.WriteInput(source, program); EmbeddedVertexFetchPlan? fetch = null; @@ -371,6 +417,23 @@ internal sealed class ShaderProgramCache throw new ShaderProgramRejectedException($"The shader resource plan is invalid: stage={source.Label} hash=0x{source.Hash:X16} shader=0x{source.Address:X16} error={exception.Message}."); } + // Descriptors read at run time index the persistent heap. Without it the plan cannot be + // bound, which is a rejected plan like any other rather than an invalid layout. + if (plan.Info.UsesRuntimeDescriptors && !_host.UsesBindlessImages) + { + const string error = "runtime image descriptors need the persistent bindless heap"; + if (dumpPlanning) ShaderPlanningDump.WriteFailure(source, error); + throw new ShaderProgramRejectedException($"The shader resource plan is invalid: stage={source.Label} hash=0x{source.Hash:X16} shader=0x{source.Address:X16} error={error}."); + } + + if (plan.Info.NullDescriptorFallbacks.Count != 0) + { + Console.Error.WriteLine( + $"[GPU][WARN][NULL_DESCRIPTOR] stage={source.Label} hash=0x{source.Hash:X16} shader=0x{source.Address:X16} " + + $"accesses={plan.Info.NullDescriptorFallbacks.Count} [{string.Join(", ", plan.Info.NullDescriptorFallbacks.Select(access => $"pc=0x{access.Pc:X} {access.Kind}"))}]: " + + "no plan-time descriptor source; these accesses read the null descriptor, so their texels read as zero."); + } + var exclusiveOr = false; foreach (var instruction in program.Instructions) { @@ -513,7 +576,8 @@ internal sealed class ShaderProgramCache { resources = ResourceMaterializer.ApplyTo(plan, specialization); layout = AllocateLayout(program, plan, resources, source.UserDataBase, (uint)source.UserData.Length, pushDataCursor, - source.Stage == ShaderStage.Compute && options.ComputeInfo!.DispatchThreadDimensions); + source.Stage == ShaderStage.Compute && options.ComputeInfo!.DispatchThreadDimensions, source.Stage, + options.Tessellation is not null || options.TessellationHull is not null); } catch (ResourcePlanException exception) { @@ -557,7 +621,7 @@ internal sealed class ShaderProgramCache var id = ++_nextProgramId; var module = _host.CreateShaderModule(compiled, source.Stage, source.Hash, id); var info = CreateProgramInfo(source, entry, resources, layout, request); - if (source.Stage == ShaderStage.Compute && + if (source.Stage == ShaderStage.Compute && options.TessellationHull is null && _host.ShaderPrewarm is { } prewarm && _codeCaptures.TryGetValue((source.Hash, source.CodeSize), out var capture)) { @@ -593,19 +657,29 @@ internal sealed class ShaderProgramCache { var enableGraphicsSubgroups = _host.GraphicsSubgroupOperationsEnabled; var sharedInt64Atomics = _host.SharedInt64AtomicsEnabled; + var exactFloat16Conversions = _host.ExactFloat16ConversionsEnabled; + var nonUniformImageIndexing = _host.NonUniformImageIndexingEnabled; + var nativeHalfConversion = _host.NativeHalfConversionExact; + var zeroOutOfBoundsReads = _host.ZeroOutOfBoundsBufferReads; var signedZeroInfNanPreserve = _host.ShaderSignedZeroInfNanPreserveFloat32Supported; switch (source.Stage) { case ShaderStage.Vertex: + case ShaderStage.TessellationEvaluation: { var info = options.VertexInfo!; return new ShaderCompileRequest(entry.Plan, resources, layout) { WaveSize = 32, + Tessellation = options.Tessellation, TraceDeviceAddressFaults = SharpEmu.HLE.GpuMemory.GuestGpuMemoryHook.TraceEnabled, ScratchDwords = info.ScratchDwords, - EnableGraphicsSubgroupOperations = enableGraphicsSubgroups, + EnableGraphicsSubgroupOperations = source.Stage == ShaderStage.TessellationEvaluation ? false : enableGraphicsSubgroups, SupportsSharedInt64Atomics = sharedInt64Atomics, + SupportsExactFloat16Conversions = exactFloat16Conversions, + SupportsNonUniformImageIndexing = nonUniformImageIndexing, + NativeHalfConversionExact = nativeHalfConversion, + ZeroOutOfBoundsBufferReads = zeroOutOfBoundsReads, ShaderSignedZeroInfNanPreserveFloat32Supported = signedZeroInfNanPreserve, RequiredVertexOutputCount = options.RequiredVertexOutputCount, VertexInputs = entry.VertexInputs, @@ -625,6 +699,8 @@ internal sealed class ShaderProgramCache case ShaderStage.Pixel: { var info = options.PixelInfo!; + if (info.EarlyDepth && info.ShaderSampleExclusionMask != 0 && !_host.PostDepthCoverageSupported) + throw SubmissionScheduler.Fatal("Sample exclusion requires post-depth coverage support."); var interpolators = new uint[info.InputCount]; Array.Copy(info.InterpolatorSettings, interpolators, interpolators.Length); return new ShaderCompileRequest(entry.Plan, resources, layout) @@ -634,8 +710,20 @@ internal sealed class ShaderProgramCache ScratchDwords = info.ScratchDwords, EnableGraphicsSubgroupOperations = enableGraphicsSubgroups, SupportsSharedInt64Atomics = sharedInt64Atomics, + SupportsExactFloat16Conversions = exactFloat16Conversions, + SupportsNonUniformImageIndexing = nonUniformImageIndexing, + NativeHalfConversionExact = nativeHalfConversion, + ZeroOutOfBoundsBufferReads = zeroOutOfBoundsReads, ShaderSignedZeroInfNanPreserveFloat32Supported = signedZeroInfNanPreserve, PixelOutputs = options.PixelOutputs, + EarlyFragmentTests = info.EarlyDepth, + PixelShaderSampleExclusionMask = info.EarlyDepth ? info.ShaderSampleExclusionMask : 0u, + PixelDepthExportEnable = info.DepthExportEnable, + PixelSampleMaskExportEnable = info.SampleMaskExportEnable, + PixelMaskExportSamples = info.MaskExportSamples, + PixelRasterizationSamples = info.RasterizationSamples, + PixelInterpolationSample = info.InterpolationSample, + PixelCustomSampleOffsets = info.CustomSampleOffsets, PixelInputEnable = options.PixelInputEnable, PixelCustomInterpolationMask = info.CustomInterpolationMask, SupportsPerVertexPixelInputs = _host.PerVertexPixelInputsSupported, @@ -645,13 +733,41 @@ internal sealed class ShaderProgramCache } default: - return BuildComputeRequest(entry.Plan, resources, layout, options.ComputeInfo!, options.ComputeSystemRegisters, - sharedInt64Atomics, _host.ExecGuardElisionEnabled, signedZeroInfNanPreserve); + { + var info = options.ComputeInfo!; + return new ShaderCompileRequest(entry.Plan, resources, layout) + { + WaveSize = info.WaveSize, + TessellationHull = options.TessellationHull, + CooperativeWave64Workgroup = options.TessellationHull is not null, + EnableExecGuardElision = info.WaveSize != 64 || _host.ExecGuardElisionEnabled, + TraceDeviceAddressFaults = SharpEmu.HLE.GpuMemory.GuestGpuMemoryHook.TraceEnabled, + ScratchDwords = info.ScratchDwords, + SupportsSharedInt64Atomics = sharedInt64Atomics, + SupportsExactFloat16Conversions = exactFloat16Conversions, + SupportsNonUniformImageIndexing = nonUniformImageIndexing, + NativeHalfConversionExact = nativeHalfConversion, + ZeroOutOfBoundsBufferReads = zeroOutOfBoundsReads, + ShaderSignedZeroInfNanPreserveFloat32Supported = signedZeroInfNanPreserve, + ComputeSystemRegisters = options.ComputeSystemRegisters, + LocalDataShareDwords = info.LocalDataShareDwords, + LocalSizeX = Math.Max(info.ThreadsX, 1), + LocalSizeY = Math.Max(info.ThreadsY, 1), + LocalSizeZ = Math.Max(info.ThreadsZ, 1), + }; + } } } + private BindingLayout AllocateLayout(Gen5ShaderProgram program, ShaderResourcePlan plan, SpecializedResourceInfo resources, + uint userDataBase, uint userDataCount, uint pushDataCursor, bool usesDispatchThreadLimits, ShaderStage stage, + bool usesTessellationData = false) => + AllocateLayout(program, plan, resources, userDataBase, userDataCount, pushDataCursor, usesDispatchThreadLimits, stage, + _host, usesTessellationData); + private static BindingLayout AllocateLayout(Gen5ShaderProgram program, ShaderResourcePlan plan, SpecializedResourceInfo resources, - uint userDataBase, uint userDataCount, uint pushDataCursor, bool usesDispatchThreadLimits) => + uint userDataBase, uint userDataCount, uint pushDataCursor, bool usesDispatchThreadLimits, ShaderStage stage, + IShaderPipelineHost host, bool usesTessellationData = false) => BindingLayout.Allocate( resources.Info, BindingLayout.CollectUserDataRegisters(program, userDataBase, userDataCount), @@ -659,19 +775,29 @@ internal sealed class ShaderProgramCache ShaderCompileRequest.RequiresFlattenedTable(plan, resources), BindingLayout.ReadsShaderBase(program), pushDataCursor, - usesDispatchThreadLimits: usesDispatchThreadLimits); + usesDispatchThreadLimits: usesDispatchThreadLimits, + usesBindlessImages: host.UsesBindlessImages, + usesRuntimeBufferStrides: host.RuntimeBufferStridesEnabled, + usesTessellationData: usesTessellationData, + // A pixel program reads its position in guest pixels; any program that samples or + // fetches an image needs to know which of them the host holds at another size. + usesRenderScale: Images.RenderScalePolicy.Enabled && + (stage == ShaderStage.Pixel || resources.Info.Images.Count != 0)); private static ShaderCompileRequest BuildComputeRequest(ShaderResourcePlan plan, SpecializedResourceInfo resources, BindingLayout layout, - ComputeInputInfo info, Gen5ComputeSystemRegisters? systemRegisters, bool sharedInt64Atomics, bool execGuardElision, - bool signedZeroInfNanPreserve) => + ComputeInputInfo info, Gen5ComputeSystemRegisters? systemRegisters, IShaderPipelineHost host) => new(plan, resources, layout) { + NativeHalfConversionExact = host.NativeHalfConversionExact, + ZeroOutOfBoundsBufferReads = host.ZeroOutOfBoundsBufferReads, WaveSize = info.WaveSize, - EnableExecGuardElision = info.WaveSize != 64 || execGuardElision, + EnableExecGuardElision = info.WaveSize != 64 || host.ExecGuardElisionEnabled, TraceDeviceAddressFaults = SharpEmu.HLE.GpuMemory.GuestGpuMemoryHook.TraceEnabled, ScratchDwords = info.ScratchDwords, - SupportsSharedInt64Atomics = sharedInt64Atomics, - ShaderSignedZeroInfNanPreserveFloat32Supported = signedZeroInfNanPreserve, + SupportsSharedInt64Atomics = host.SharedInt64AtomicsEnabled, + SupportsExactFloat16Conversions = host.ExactFloat16ConversionsEnabled, + SupportsNonUniformImageIndexing = host.NonUniformImageIndexingEnabled, + ShaderSignedZeroInfNanPreserveFloat32Supported = host.ShaderSignedZeroInfNanPreserveFloat32Supported, ComputeSystemRegisters = systemRegisters, LocalDataShareDwords = info.LocalDataShareDwords, LocalSizeX = Math.Max(info.ThreadsX, 1), @@ -680,8 +806,7 @@ internal sealed class ShaderProgramCache }; internal static bool TryCompilePrewarm(ComputePrewarmRecord record, ShaderCodeCapture code, IGuestGpuBackend compiler, - bool sharedInt64Atomics, bool execGuardElision, bool signedZeroInfNanPreserve, out IGuestCompiledShader? compiled, - out BindingLayout? layout, out string error) + IShaderPipelineHost host, out IGuestCompiledShader? compiled, out BindingLayout? layout, out string error) { compiled = null; layout = null; @@ -696,9 +821,9 @@ internal sealed class ShaderProgramCache waveSize: record.Info.WaveSize); var resources = ResourceMaterializer.ApplyTo(plan, record.Specialization); layout = AllocateLayout(program, plan, resources, record.UserDataBase, record.UserDataCount, record.PushDataCursor, - record.Info.DispatchThreadDimensions); + record.Info.DispatchThreadDimensions, ShaderStage.Compute, host); var request = BuildComputeRequest(plan, resources, layout, record.Info, record.SystemRegisters, - sharedInt64Atomics, execGuardElision, signedZeroInfNanPreserve); + host); return compiler.TryCompileProgram(request, out compiled, out error) && compiled is not null; } catch (Exception exception) @@ -708,7 +833,7 @@ internal sealed class ShaderProgramCache } } - // A fill kernel computes its thread index, moves one constant into a VGPR and stores it once. + // Identify a constant store value without making assumptions about its address. internal static uint? FindConstantStoreValue(Gen5ShaderProgram program) { var constants = new Dictionary(); @@ -832,9 +957,12 @@ internal sealed class ShaderProgramCache { ShaderStage.Vertex => ShaderStageKind.Vertex, ShaderStage.Pixel => ShaderStageKind.Pixel, + ShaderStage.TessellationEvaluation => ShaderStageKind.TessellationEvaluation, _ => ShaderStageKind.Compute, }, Hash = source.Hash, + TessellationFactorBuffer = request.TessellationHull is { } hull + ? checked((int)Gen5TessellationLowering.FindFactorBuffer(entry.Plan, hull.HullEntryPc)) : -1, UserDataBase = source.UserDataBase, UserDataCount = (uint)source.UserData.Length, ParameterExportMask = entry.Program.ParameterExportMask, @@ -843,6 +971,7 @@ internal sealed class ShaderProgramCache InstanceOffsetScalarRegister = entry.EmbeddedFetch?.InstanceOffsetScalarRegister ?? ShaderProgramInfo.NoScalarRegister, UsesDeviceAddresses = info.UsesDeviceAddresses, ConstantStoreValue = FindConstantStoreValue(entry.Program), + ImmediateConstantFill = ConstantFillDetector.DetectImmediate(entry.Program), HasBitwiseExclusiveOr = entry.HasBitwiseExclusiveOr, ConstantFill = entry.ConstantFill, BoundedFill = entry.BoundedFill, diff --git a/src/SharpEmu.Libs/Gpu/Pipelines/StageStaticKey.cs b/src/SharpEmu.Libs/Gpu/Pipelines/StageStaticKey.cs index b853bb9..d92498f 100644 --- a/src/SharpEmu.Libs/Gpu/Pipelines/StageStaticKey.cs +++ b/src/SharpEmu.Libs/Gpu/Pipelines/StageStaticKey.cs @@ -72,10 +72,20 @@ public static class StageStaticKey key.Add(Bit(info.KillEnable)); key.Add(Bit(info.DepthExportEnable)); key.Add(Bit(info.SampleMaskExportEnable)); - key.Add(Bit(info.EarlyDepth)); - foreach (var mode in info.TargetOutputModes) + key.Add(info.SampleMaskExportEnable ? info.MaskExportSamples : 0u); + key.Add(info.SampleMaskExportEnable ? info.RasterizationSamples : 0u); + key.Add(info.InterpolationSample ?? uint.MaxValue); + key.Add((uint)info.CustomSampleOffsets.Count); + foreach (var offset in info.CustomSampleOffsets) { - key.Add(mode); + key.Add(BitConverter.SingleToUInt32Bits(offset.X)); + key.Add(BitConverter.SingleToUInt32Bits(offset.Y)); + } + key.Add(Bit(info.EarlyDepth)); + key.Add(info.EarlyDepth ? info.ShaderSampleExclusionMask : 0u); + for (var index = 0; index < info.TargetOutputModes.Length; index++) + { + key.Add(info.TargetOutputModes[index] | (uint)info.TargetExportFormats[index] << 8); } for (var first = 0; first < PixelInputInfo.TargetCount; first += 4) diff --git a/src/SharpEmu.Libs/Gpu/Pipelines/VertexInputResolver.cs b/src/SharpEmu.Libs/Gpu/Pipelines/VertexInputResolver.cs index 4265201..df493cc 100644 --- a/src/SharpEmu.Libs/Gpu/Pipelines/VertexInputResolver.cs +++ b/src/SharpEmu.Libs/Gpu/Pipelines/VertexInputResolver.cs @@ -360,7 +360,24 @@ public static class VertexInputResolver for (var bufferIndex = 0; bufferIndex < buffers.Count; bufferIndex++) { var buffer = buffers[bufferIndex]; - result[bufferIndex] = new VertexInputBuffer(buffer.Address, buffer.Stride, buffer.RecordCount, buffer.FetchIndex != 0); + uint minimumFetchBytes = 0; + foreach (var resourceIndex in buffer.AttributeIndices) + { + var resource = resources[resourceIndex]; + var descriptor = resource.Descriptor; + // OOB_SELECT=2 only tests for an unbound (zero-record) descriptor. + // A constant stream still reads the whole formatted element, even + // when NUM_RECORDS is smaller than that element's byte width. + if (descriptor.Stride == 0 && descriptor.OutOfBounds == 2 && descriptor.RecordCount != 0 && + VertexAttributeFormats.TryResolve(in descriptor, 4, out var format, out _)) + { + minimumFetchBytes = Math.Max(minimumFetchBytes, Images.ViewFormatRules.BlockBytes(format)); + } + } + result[bufferIndex] = new VertexInputBuffer(buffer.Address, buffer.Stride, buffer.RecordCount, buffer.FetchIndex != 0) + { + MinimumFetchBytes = minimumFetchBytes, + }; foreach (var resourceIndex in buffer.AttributeIndices) { var resource = resources[resourceIndex]; diff --git a/src/SharpEmu.Libs/Gpu/Rendering/ColorTargetResolver.cs b/src/SharpEmu.Libs/Gpu/Rendering/ColorTargetResolver.cs index 3e5e630..32d916e 100644 --- a/src/SharpEmu.Libs/Gpu/Rendering/ColorTargetResolver.cs +++ b/src/SharpEmu.Libs/Gpu/Rendering/ColorTargetResolver.cs @@ -29,6 +29,13 @@ public static class ColorTargetResolver public static ColorTargetResolution? Resolve(ContextRegisters context, uint slot, uint drawLayerOffset, bool ignoreTargetMask, out uint resolvedSlot) { resolvedSlot = slot == FirstBoundSlot ? FirstBound(context) : slot; + // Disabling CB takes precedence over stale target addresses and masks. + // Depth-only passes may leave the preceding color targets bound. + if (context.ColorControl.Mode == 0) + { + return null; + } + return ImageRequestBuilders.ColorTarget(in context.ColorTargets[resolvedSlot], context.RenderTargetMaskForSlot(resolvedSlot), drawLayerOffset, ignoreTargetMask); } } diff --git a/src/SharpEmu.Libs/Gpu/Rendering/DepthTargetResolver.cs b/src/SharpEmu.Libs/Gpu/Rendering/DepthTargetResolver.cs index c60456a..56b0b2d 100644 --- a/src/SharpEmu.Libs/Gpu/Rendering/DepthTargetResolver.cs +++ b/src/SharpEmu.Libs/Gpu/Rendering/DepthTargetResolver.cs @@ -76,6 +76,20 @@ public readonly record struct DepthStencilState( : ImageLayout.DepthAttachmentOptimal; } + // A sampled attachment needs the read-only layouts; a draw that writes needs the writable + // one. A draw that neither samples nor writes keeps the read-only layout it is already in, + // so sampling and depth-testing draws can alternate inside one rendering scope. + public static ImageLayout AttachmentLayoutFor(DepthStencilState state, Format format, bool sampled, ImageLayout? current) + { + var readOnly = state.AttachmentLayout(format); + if (sampled) + { + return readOnly; + } + + return state.IsReadOnly(format) && current == readOnly ? readOnly : WritableAttachmentLayout(format); + } + public ImageLayout AttachmentLayout(Format format) { var available = ViewFormatRules.DepthAspects(format); @@ -226,6 +240,23 @@ public static class DepthTargetResolver private static StencilOperations ConvertFace(byte fail, byte pass, byte depthFail, CompareOp compare, byte operationValue, ref StencilMasks masks, Func fatal) { + if (masks.WriteMask != 0 && compare == CompareOp.Equal && + (masks.WriteMask & ~masks.CompareMask) == 0 && fail == 0 && + (pass == 0 || pass == ReplaceWithOperationValue) && + (depthFail == 0 || depthFail == ReplaceWithOperationValue) && + (pass == ReplaceWithOperationValue || depthFail == ReplaceWithOperationValue)) + { + // The equal test fixes every written bit. Toggle only differing bits + // to produce the replacement without changing the comparison reference. + var invertMask = masks.WriteMask & (masks.Reference ^ operationValue); + masks = masks with { WriteMask = invertMask }; + return new StencilOperations( + StencilOp.Keep, + pass == ReplaceWithOperationValue && invertMask != 0 ? StencilOp.Invert : StencilOp.Keep, + depthFail == ReplaceWithOperationValue && invertMask != 0 ? StencilOp.Invert : StencilOp.Keep, + compare); + } + ReadOnlySpan operations = [fail, pass, depthFail]; Span converted = stackalloc StencilOp[3]; var reference = masks.Reference; @@ -239,7 +270,7 @@ public static class DepthTargetResolver } var replacement = operations[index] == ReplaceWithOperationValue ? operationValue : masks.Reference; - // Vulkan shares one reference between comparison and all replacements on this face. + // Vulkan shares one reference between comparison and every replacement on this face. if (((reference ^ replacement) & requiredBits & masks.WriteMask) != 0) { throw fatal($"The stencil replacement references conflict: compare={compare} compareMask=0x{masks.CompareMask:X2} " + diff --git a/src/SharpEmu.Libs/Gpu/Rendering/DrawTargets.cs b/src/SharpEmu.Libs/Gpu/Rendering/DrawTargets.cs index 0b901da..3912c01 100644 --- a/src/SharpEmu.Libs/Gpu/Rendering/DrawTargets.cs +++ b/src/SharpEmu.Libs/Gpu/Rendering/DrawTargets.cs @@ -15,13 +15,23 @@ public struct ColorTargetState public ResourceSlotIdentifier Image; public ImageView View; - public ColorTargetState(in ColorTargetResolution resolution, uint slot, ResourceSlotIdentifier image) + // The host resolution the cache actually gave this image, which is one once the image + // has been dropped back to guest resolution. Asking the scaling rules again would miss + // that, so the render area, viewport and scissor read it from here. + public float RenderScale; + + public ColorTargetState(in ColorTargetResolution resolution, uint slot, ResourceSlotIdentifier image, float renderScale) { Resolution = resolution; Slot = slot; Image = image; View = default; + RenderScale = renderScale; } + + public readonly Extent2D HostExtent => new( + RenderScalePolicy.ScaleLength(Resolution.Extent.Width, RenderScale), + RenderScalePolicy.ScaleLength(Resolution.Extent.Height, RenderScale)); } // The depth target of a draw; HasTarget is false when no depth or stencil state is active. @@ -32,14 +42,22 @@ public struct DepthAttachmentState public ImageView View; public bool MetadataClear; - public DepthAttachmentState(in DepthTargetState target, ResourceSlotIdentifier image) + // See ColorTargetState.RenderScale. + public float RenderScale; + + public DepthAttachmentState(in DepthTargetState target, ResourceSlotIdentifier image, float renderScale) { Target = target; Image = image; View = default; MetadataClear = false; + RenderScale = renderScale; } + public readonly uint HostWidth => RenderScalePolicy.ScaleLength(Target.Target.Width, RenderScale); + + public readonly uint HostHeight => RenderScalePolicy.ScaleLength(Target.Target.Height, RenderScale); + public readonly bool HasTarget => Image.IsValid && Target.Target.Format != Format.Undefined; // The attachment loads a clear when the registers or the metadata say so. diff --git a/src/SharpEmu.Libs/Gpu/Rendering/IRenderHost.cs b/src/SharpEmu.Libs/Gpu/Rendering/IRenderHost.cs index a4ca4e3..2a5965a 100644 --- a/src/SharpEmu.Libs/Gpu/Rendering/IRenderHost.cs +++ b/src/SharpEmu.Libs/Gpu/Rendering/IRenderHost.cs @@ -4,6 +4,7 @@ using SharpEmu.Libs.Gpu.Buffers; using SharpEmu.Libs.Gpu.Images; using SharpEmu.Libs.Gpu.Scheduling; +using SharpEmu.HLE.GpuMemory; using Silk.NET.Vulkan; namespace SharpEmu.Libs.Gpu.Rendering; @@ -63,7 +64,9 @@ public readonly record struct DynamicDrawState( StencilMasks FrontStencil, StencilMasks BackStencil, uint ColorWriteCount, - byte ColorWriteEnableMask); + byte ColorWriteEnableMask, + float DepthMinBounds = 0f, + float DepthMaxBounds = 1f); // The host-side descriptors of one shader stage, prepared before the draw or dispatch records. public interface IPreparedBindings @@ -92,6 +95,9 @@ public interface IRenderHost bool TryReadGuest(ulong address, Span destination); + // Reads guest bytes only when no pending GPU buffer or image write owns them. + bool TryReadCleanGuestBytes(ulong address, Span destination); + // The part of the range that is mapped from its start; zero when the start is unmapped. ulong ClampMappedSize(ulong address, ulong size); @@ -99,6 +105,13 @@ public interface IRenderHost void BindRenderTarget(ResourceSlotIdentifier image); + // The attachments of one pass must share a host resolution; a disagreement drops the + // scaled image back to guest resolution for good and the draw resolves its targets again. + void DemoteRenderScale(ResourceSlotIdentifier image); + + // The host resolution multiplier of one cached image; one when it is at guest resolution. + float GetRenderScale(ResourceSlotIdentifier image); + void ResetBindings(); // Finds the image again when it changed, makes the view, sets the layout and resolves a metadata clear. @@ -108,10 +121,18 @@ public interface IRenderHost DepthAttachmentAcquisition AcquireDepthAttachment(in DepthAttachmentState depth); void TransitionDepthAttachment(in DepthAttachmentState depth, ImageLayout layout, ImageAspectFlags writeAspects); + bool NativeTwoSampleMixedSupported => false; + void ConfigureDepthSampleLocations(in DepthAttachmentState depth, ReadOnlySpan words, ImageLayout layout, ImageAspectFlags writeAspects) + => TransitionDepthAttachment(in depth, layout, writeAspects); BufferBinding NullBuffer { get; } BufferBinding ObtainBuffer(ulong address, ulong size, bool isWritten); + ulong ObtainBufferDeviceAddress(ulong address, ulong size, bool isWritten) => + throw new NotSupportedException("The render host does not expose buffer addresses for tessellation."); + + // Resolve allocations that can overlap before descriptors or geometry capture their handles. + void PrepareBufferAllocations(ReadOnlySpan ranges); // Copies host bytes into the stream ring for the current recording. BufferBinding UploadTransient(ReadOnlySpan data, uint alignment); @@ -124,6 +145,8 @@ public interface IRenderHost IResourcePreparation BeginPreparation(); IPreparedBindings PrepareBindings(ShaderStageResources stage); + void UpdateTessellationData(IPreparedBindings prepared, uint[] data) => + throw new NotSupportedException("The render host does not support tessellation runtime data."); void PrepareDeviceAddresses(); @@ -140,9 +163,13 @@ public interface IRenderHost // Whether a texture bound for the draw being prepared reads the depth attachment's subresources. bool SamplesDepthAttachment(in DepthAttachmentState depth) => true; + // The layout the depth attachment's subresources are in now, or null when unknown or mixed. + ImageLayout? DepthAttachmentLayout(in DepthAttachmentState depth) => null; + // The next draw stores to buffers or storage images; called before its BeginRendering. void PrepareMemoryWritingDraw() { } + void PrepareGraphicsPipeline(in PipelineHandle pipeline) { } void BindPipeline(PipelineBindPoint bindPoint, in PipelineHandle pipeline); void Draw(uint vertexCount, uint instanceCount, uint firstVertex, uint firstInstance); @@ -153,6 +180,10 @@ public interface IRenderHost void DrawIndexedIndirect(BufferBinding arguments) => throw new NotSupportedException("The render host does not draw from indirect arguments."); + // One draw whose counts the GPU reads from the buffer (VkDrawIndirectCommand layout). + void DrawIndirect(BufferBinding arguments) => + throw new NotSupportedException("The render host does not draw from indirect arguments."); + void Dispatch(uint groupsX, uint groupsY, uint groupsZ); // Executes a dispatch from three uint32 group counts in guest memory when the backend can diff --git a/src/SharpEmu.Libs/Gpu/Rendering/IShaderPipelineProvider.cs b/src/SharpEmu.Libs/Gpu/Rendering/IShaderPipelineProvider.cs index dce8bc5..ee4bd44 100644 --- a/src/SharpEmu.Libs/Gpu/Rendering/IShaderPipelineProvider.cs +++ b/src/SharpEmu.Libs/Gpu/Rendering/IShaderPipelineProvider.cs @@ -23,6 +23,7 @@ public readonly record struct VertexPositionStream(ulong Address, uint Stride, u public sealed class GraphicsPrograms { + public TessellationDrawPrograms? Tessellation { get; init; } public ShaderProgram Vertex { get; init; } public ShaderProgram Pixel { get; init; } public VertexInputInfo VertexInput { get; init; } = new(); @@ -39,6 +40,9 @@ public sealed class GraphicsPrograms public bool DisableBlending { get; init; } } +public sealed record TessellationDrawPrograms(SharpEmu.ShaderCompiler.Gen5TessellationInfo Configuration, + SharpEmu.ShaderCompiler.Gen5TessellationHullInfo HullConfiguration, ComputeProgram Hull); + public sealed class ComputeProgram { public ShaderProgram Program { get; init; } diff --git a/src/SharpEmu.Libs/Gpu/Rendering/RenderExecutor.Compute.cs b/src/SharpEmu.Libs/Gpu/Rendering/RenderExecutor.Compute.cs index bf29ca8..244390b 100644 --- a/src/SharpEmu.Libs/Gpu/Rendering/RenderExecutor.Compute.cs +++ b/src/SharpEmu.Libs/Gpu/Rendering/RenderExecutor.Compute.cs @@ -1,8 +1,10 @@ // Copyright (C) 2026 SharpEmu Emulator Project // SPDX-License-Identifier: GPL-2.0-or-later +using System.Buffers.Binary; using SharpEmu.Libs.Gpu.GpuCommands.Registers; using SharpEmu.Libs.Gpu.Scheduling; +using SharpEmu.ShaderCompiler.Resources; using SharpEmu.ShaderCompiler.Vulkan; using Silk.NET.Vulkan; using ResourceSnapshot = SharpEmu.ShaderCompiler.Resources.ResourceSnapshot; @@ -13,6 +15,7 @@ public readonly record struct ComputeImageClear(BufferDescriptorWords Descriptor public sealed partial class RenderExecutor { + private const uint DispatchInitiatorUseThreadDimensions = 1u << 5; private const uint DispatchInitiatorBaseBits = 0x41; private const uint DispatchInitiatorModifierBits = 0xA038; @@ -51,6 +54,35 @@ public sealed partial class RenderExecutor } var useThreadDimensions = (dispatchInitiator & DispatchInitiatorUseThreadDimensions) != 0; + // A zero dimension executes no work. Check before materializing shaders: an indirect + // buffer can be read here only when no pending GPU write can change its contents. + var zeroDispatch = false; + if (indirectArgumentsAddress == 0 || useThreadDimensions) + { + zeroDispatch = groupsX == 0 || groupsY == 0 || groupsZ == 0; + } + else + { + Span arguments = stackalloc byte[3 * sizeof(uint)]; + if (_host.TryReadCleanGuestBytes(indirectArgumentsAddress, arguments)) + { + zeroDispatch = BinaryPrimitives.ReadUInt32LittleEndian(arguments) == 0 || + BinaryPrimitives.ReadUInt32LittleEndian(arguments[sizeof(uint)..]) == 0 || + BinaryPrimitives.ReadUInt32LittleEndian(arguments[(2 * sizeof(uint))..]) == 0; + } + } + + if (zeroDispatch) + { + if (RenderTrace.Enabled && RenderTrace.ZeroDispatch()) + { + RenderTrace.Write($"Skipping a zero-sized dispatch: groups={groupsX}x{groupsY}x{groupsZ} " + + $"indirect=0x{indirectArgumentsAddress:X16} initiator=0x{dispatchInitiator:X8} shader=0x{compute.Address:X16}"); + } + + return; + } + var computeProgram = _pipelines.GetComputeProgram(compute, banks.Context.ShaderInterface, dispatchInitiator, groupsX, groupsY, groupsZ); if (computeProgram.Consumed) { @@ -132,16 +164,6 @@ public sealed partial class RenderExecutor } } - if (indirectArgumentsAddress == 0 && (groupsX == 0 || groupsY == 0 || groupsZ == 0)) - { - if (RenderTrace.Enabled && RenderTrace.ZeroDispatch()) - { - RenderTrace.Write($"Skipping a zero-sized dispatch: groups={groupsX}x{groupsY}x{groupsZ} initiator=0x{dispatchInitiator:X8} shader=0x{compute.Address:X16}"); - } - - return; - } - _host.EndRendering(); using (_host.BeginPreparation()) { @@ -228,6 +250,13 @@ public sealed partial class RenderExecutor return BufferDescriptorWords.From(words); } + // The program's packed stride against the descriptor's. With runtime strides the + // program carries only the flags and indexes with the descriptor's stride. + private static bool StrideMatches(uint programStride, uint descriptorStride) => + (programStride & ~BufferSpecialization.StrideMask) == (descriptorStride & ~BufferSpecialization.StrideMask) && + ((programStride & BufferSpecialization.StrideMask) == 0 || + (programStride & BufferSpecialization.StrideMask) == (descriptorStride & BufferSpecialization.StrideMask)); + // A full overwrite of registered metadata by a compute shader becomes a tracked clear. private bool TryConsumeMetadataClear(ComputeInputInfo input) { @@ -283,7 +312,9 @@ public sealed partial class RenderExecutor { var program = input.Stage.Program ?? throw _host.Fatal("The compute stage has no program."); var resources = input.Stage.Resources; - if (program.ConstantStoreValue is not { } value || + if (program.ConstantStoreValue is not { } value || program.ImmediateConstantFill is not { } fill || + program.UserDataBase != 0 || fill.GroupScalarRegister != (uint)input.WorkgroupRegister || + fill.DestinationScalarResource + 4 > resources.UserData.Length || program.Buffers.Length != 1 || resources.Buffers.Length != 1 || resources.Buffers[0].Length != 4 || program.Images.Length != 0 || program.SamplerCount != 0 || program.UsesDeviceAddresses) { @@ -292,6 +323,8 @@ public sealed partial class RenderExecutor var info = program.Buffers[0]; var descriptor = BufferDescriptorWords.From(resources.Buffers[0]); + if (!resources.UserData.AsSpan((int)fill.DestinationScalarResource, 4).SequenceEqual(resources.Buffers[0])) + return null; var threads = (ulong)groupsX * input.ThreadsX; if (!info.Formatted || !info.Written || info.Read || info.Atomic || info.Scalar || info.MaxByteExtent != sizeof(uint) || descriptor.Stride != sizeof(uint) || descriptor.Format != BufferDescriptorWords.Format32UInt || descriptor.SwizzleEnabled || @@ -314,7 +347,10 @@ public sealed partial class RenderExecutor { var program = input.Stage.Program ?? throw _host.Fatal("The compute stage has no program."); var resources = input.Stage.Resources; - if (program.Buffers.Length != 2 || resources.Buffers.Length != 2 || program.Images.Length != 0 || program.SamplerCount != 0 || + if (program.ConstantFill is not { } fill || program.UserDataBase != 0 || + fill.GroupScalarRegister != (uint)input.WorkgroupRegister || + fill.DestinationScalarResource + 4 > resources.UserData.Length || fill.SourceScalarResource + 4 > resources.UserData.Length || + program.Buffers.Length != 2 || resources.Buffers.Length != 2 || program.Images.Length != 0 || program.SamplerCount != 0 || program.UsesDeviceAddresses || resources.Images.Length != 0 || resources.Samplers.Length != 0 || resources.Buffers[0].Length != 4 || resources.Buffers[1].Length != 4) { @@ -342,6 +378,9 @@ public sealed partial class RenderExecutor var descriptor = BufferDescriptorWords.From(resources.Buffers[target]); var valueDescriptor = BufferDescriptorWords.From(resources.Buffers[source]); + if (!resources.UserData.AsSpan((int)fill.DestinationScalarResource, 4).SequenceEqual(resources.Buffers[target]) || + !resources.UserData.AsSpan((int)fill.SourceScalarResource, 4).SequenceEqual(resources.Buffers[source])) + return null; var threads = (ulong)groupsX * input.ThreadsX; if (descriptor.Stride != sizeof(uint) || descriptor.Format != BufferDescriptorWords.Format32UInt || descriptor.SwizzleEnabled || descriptor.IndexStride != 0 || descriptor.AddThreadId || descriptor.RecordCount == 0 || @@ -384,7 +423,7 @@ public sealed partial class RenderExecutor var descriptor = BufferDescriptorWords.From(words); if (!resource.Formatted || !resource.Written || resource.Read || resource.Atomic || resource.Scalar || resource.MaxByteExtent != ImageClearStride || descriptor.Stride != ImageClearStride || descriptor.Format != BufferDescriptorWords.Format32x4UInt || descriptor.SwizzleEnabled || - descriptor.IndexStride != 0 || descriptor.AddThreadId || resource.PackedStride != descriptor.PackedStride || + descriptor.IndexStride != 0 || descriptor.AddThreadId || !StrideMatches(resource.PackedStride, descriptor.PackedStride) || program.UserDataBase != 0 || resources.UserData.Length != ImageClearUserDataCount) { return null; diff --git a/src/SharpEmu.Libs/Gpu/Rendering/RenderExecutor.DynamicState.cs b/src/SharpEmu.Libs/Gpu/Rendering/RenderExecutor.DynamicState.cs index cc31c82..ce973bc 100644 --- a/src/SharpEmu.Libs/Gpu/Rendering/RenderExecutor.DynamicState.cs +++ b/src/SharpEmu.Libs/Gpu/Rendering/RenderExecutor.DynamicState.cs @@ -2,6 +2,7 @@ // SPDX-License-Identifier: GPL-2.0-or-later using SharpEmu.Libs.Gpu.GpuCommands.Registers; +using SharpEmu.Libs.Gpu.Images; using SharpEmu.Libs.VideoOut; using Silk.NET.Vulkan; @@ -28,6 +29,18 @@ public sealed partial class RenderExecutor return hostDepthBits == 0 ? guestFactor : MathF.ScaleB(guestFactor, hostDepthBits + offset.NegativeDepthBits); } + // The host resolution of the attachments a draw writes; all of them share one rule, so + // the first bound target decides. + private static float AttachmentRenderScale(in DrawState state) + { + if (state.ColorCount > 0 && state.Colors[0].Image.IsValid) + { + return state.Colors[0].RenderScale; + } + + return state.Depth.Image.IsValid ? state.Depth.RenderScale : 1f; + } + // The dynamic state of a draw from the context bank and the resolved targets. private DynamicDrawState BuildDynamicState(ContextRegisters context, in DrawState state) { @@ -36,6 +49,9 @@ public sealed partial class RenderExecutor var limits = _host.Limits; uint framebufferWidth; uint framebufferHeight; + // The guest describes its viewport and scissor in guest pixels; the attachments decide + // how many host texels one of those covers. + var renderScale = AttachmentRenderScale(in state); if (state.ColorCount > 0 && state.Colors[0].Image.IsValid) { framebufferWidth = state.Colors[0].Resolution.Extent.Width; @@ -53,6 +69,13 @@ public sealed partial class RenderExecutor } var scissor = ResolveScissor(viewportRegisters, context.ScanMode, framebufferWidth, framebufferHeight); + if (renderScale != 1f) + { + scissor = ScaleScissor(scissor, renderScale, + RenderScalePolicy.ScaleLength(framebufferWidth, renderScale), + RenderScalePolicy.ScaleLength(framebufferHeight, renderScale)); + } + ref readonly var viewport = ref viewportRegisters.Viewports[0]; float viewportX; float viewportY; @@ -60,17 +83,19 @@ public sealed partial class RenderExecutor float viewportHeight; if (context.Clip.ClipDisable) { + // The vertex program bakes the transform onto the same reference extent, so + // multiplying only the host viewport puts guest pixels at host positions. viewportX = 0; viewportY = 0; - viewportWidth = Math.Min(limits.MaxViewportWidth, MaxViewportDimension); - viewportHeight = Math.Min(limits.MaxViewportHeight, MaxViewportDimension); + viewportWidth = RenderScalePolicy.ClipSpaceReferenceExtent(Math.Min(limits.MaxViewportWidth, MaxViewportDimension)) * renderScale; + viewportHeight = RenderScalePolicy.ClipSpaceReferenceExtent(Math.Min(limits.MaxViewportHeight, MaxViewportDimension)) * renderScale; } else { - viewportX = viewport.XOffset - viewport.XScale; - viewportY = viewport.YOffset - viewport.YScale; - viewportWidth = viewport.XScale * 2f; - viewportHeight = viewport.YScale * 2f; + viewportX = (viewport.XOffset - viewport.XScale) * renderScale; + viewportY = (viewport.YOffset - viewport.YScale) * renderScale; + viewportWidth = viewport.XScale * 2f * renderScale; + viewportHeight = viewport.YScale * 2f * renderScale; } var lineWidth = context.LineWidth; @@ -136,7 +161,24 @@ public sealed partial class RenderExecutor depthState.FrontMasks, depthState.BackMasks, state.ColorCount, - colorWriteMask); + colorWriteMask, + depthState.DepthMinBounds, + depthState.DepthMaxBounds); + } + + // A guest scissor in host texels. The edges round outward so no guest pixel the game + // asked for loses its host coverage, and the result stays inside the attachment. + public static ScissorRectangle ScaleScissor(ScissorRectangle scissor, float scale, uint hostWidth, uint hostHeight) + { + static int Floor(int value, float scale) => (int)MathF.Floor(value * scale); + static int Ceiling(int value, float scale) => (int)MathF.Ceiling(value * scale); + var left = Math.Clamp(Floor(scissor.Left, scale), 0, (int)hostWidth); + var top = Math.Clamp(Floor(scissor.Top, scale), 0, (int)hostHeight); + return new ScissorRectangle( + left, + top, + Math.Clamp(Ceiling(scissor.Right, scale), left, (int)hostWidth), + Math.Clamp(Ceiling(scissor.Bottom, scale), top, (int)hostHeight)); } // The screen, window, generic, viewport and clip rectangles intersected and clamped to the extent. diff --git a/src/SharpEmu.Libs/Gpu/Rendering/RenderExecutor.Resources.cs b/src/SharpEmu.Libs/Gpu/Rendering/RenderExecutor.Resources.cs index 5831fae..7c3b71e 100644 --- a/src/SharpEmu.Libs/Gpu/Rendering/RenderExecutor.Resources.cs +++ b/src/SharpEmu.Libs/Gpu/Rendering/RenderExecutor.Resources.cs @@ -4,6 +4,7 @@ using SharpEmu.Libs.Gpu.GpuCommands.Registers; using SharpEmu.Libs.Gpu.Images; using SharpEmu.Libs.Gpu.Scheduling; +using SharpEmu.HLE.GpuMemory; using SharpEmu.Libs.VideoOut; using Silk.NET.Vulkan; @@ -25,6 +26,26 @@ public sealed partial class RenderExecutor private readonly record struct PreparedIndexBuffer(BufferBinding Binding, ulong Size, IndexType Type); + private void PrepareDrawBufferAllocations(VertexInputInfo vertexInput, in IndexSource index, ulong indirectAddress) + { + if (vertexInput.Buffers.Length > VertexInputInfo.MaxBuffers) + throw _host.Fatal($"The vertex input has too many buffers: count={vertexInput.Buffers.Length} max={VertexInputInfo.MaxBuffers}."); + Span ranges = stackalloc GuestSpan[VertexInputInfo.MaxBuffers + 2]; + var count = 0; + foreach (var vertex in vertexInput.Buffers) + { + if (vertex.Size == 0) continue; + if (vertex.Address == 0 || vertex.Size > ulong.MaxValue - vertex.Address) + throw _host.Fatal($"The vertex buffer range is invalid: address=0x{vertex.Address:X16} size=0x{vertex.Size:X16}."); + ranges[count++] = new GuestSpan(vertex.Address, _host.ClampMappedSize(vertex.Address, vertex.Size)); + } + if (index.Enabled && index.HostData is null) + ranges[count++] = new GuestSpan(index.Address, index.Size); + if (indirectAddress != 0) + ranges[count++] = new GuestSpan(indirectAddress, IndexedIndirectArgumentsSize); + _host.PrepareBufferAllocations(ranges[..count]); + } + // Merges the vertex ranges, obtains one host buffer per merged range and offsets every slot into it. private BufferBinding[] AcquireVertexBuffers(VertexInputInfo vertexInput) { @@ -217,8 +238,13 @@ public sealed partial class RenderExecutor IPreparedBindings? pixelBindings; try { - vertexBindings = PrepareBindings(vertexInput.Stage); - pixelBindings = state.PixelActive ? PrepareBindings(pixelInput.Stage) : null; + // The pixel program reads its position in host texels; the attachments say how + // many of those one guest pixel covers. + var attachmentScale = AttachmentRenderScale(in state); + vertexBindings = PrepareBindings(vertexInput.Stage with { AttachmentRenderScale = attachmentScale }); + pixelBindings = state.PixelActive + ? PrepareBindings(pixelInput.Stage with { AttachmentRenderScale = attachmentScale }) + : null; } catch (DrawImageTypeMismatchException rejection) { @@ -237,6 +263,11 @@ public sealed partial class RenderExecutor } var vertexProgram = vertexInput.Stage.Program ?? throw _host.Fatal("The vertex stage has no program."); var pixelProgram = pixelBindings is null ? null : pixelInput.Stage.Program ?? throw _host.Fatal("The pixel stage has no program."); + DropUnwrittenColorTargets(context, ref state, pixelProgram); + state.Rendering = AcquireAttachments(ref state, context); + // Attachment uploads can submit work. Finish every allocation merge before any + // shader descriptor, vertex binding or index binding takes a buffer handle. + PrepareDrawBufferAllocations(vertexInput, in indexSource, emission.IndirectArgumentsAddress); if (vertexProgram.UsesDeviceAddresses || (pixelProgram?.UsesDeviceAddresses ?? false)) { _host.PrepareDeviceAddresses(); @@ -251,10 +282,8 @@ public sealed partial class RenderExecutor var vertexBuffers = AcquireVertexBuffers(vertexInput); var indexBuffer = AcquireIndexBuffer(in indexSource); var indirectArguments = emission.IndirectArgumentsAddress != 0 - ? _host.ObtainBuffer(emission.IndirectArgumentsAddress, IndexedIndirectArgumentsSize, isWritten: false) + ? _host.ObtainBuffer(emission.IndirectArgumentsAddress, emission.Indexed ? IndexedIndirectArgumentsSize : AutoIndirectArgumentsSize, isWritten: false) : default; - DropUnwrittenColorTargets(context, ref state, pixelProgram); - state.Rendering = AcquireAttachments(ref state); // Nothing after the pipeline touches guest memory. var pipeline = _pipelines.CreateGraphicsPipeline( BoundColors(ref state), @@ -303,6 +332,7 @@ public sealed partial class RenderExecutor _host.PrepareMemoryWritingDraw(); } + _host.PrepareGraphicsPipeline(in pipeline); _host.BeginRendering(in state.Rendering); _host.BindPipeline(PipelineBindPoint.Graphics, in pipeline); if (setAutoDebug) @@ -310,11 +340,22 @@ public sealed partial class RenderExecutor SetDrawDebugPhase(submitId, in draw, 0x500); } - if (emission.IndirectArgumentsAddress != 0) + if (state.Programs.Tessellation is not null) + { + _host.Draw(draw.Count, 1, 0, 0); + } + else if (emission.IndirectArgumentsAddress != 0) { // Uploads and shader writes end with barriers to all commands, so the // indirect read sees them. - _host.DrawIndexedIndirect(indirectArguments); + if (emission.Indexed) + { + _host.DrawIndexedIndirect(indirectArguments); + } + else + { + _host.DrawIndirect(indirectArguments); + } } else { @@ -328,7 +369,7 @@ public sealed partial class RenderExecutor var writeStages = PipelineStageFlags.None; if (HasBufferWrites(vertexInput.Stage)) { - writeStages |= PipelineStageFlags.VertexShaderBit; + writeStages |= vertexInput.Tessellation is null ? PipelineStageFlags.VertexShaderBit : PipelineStageFlags.TessellationEvaluationShaderBit; } if (state.PixelActive && HasBufferWrites(pixelInput.Stage)) @@ -378,6 +419,12 @@ public sealed partial class RenderExecutor throw _host.Fatal($"The primitive type is unknown for an indexed draw: primitiveType={userConfig.PrimitiveType}."); } + if (IsLegacyRectangleBatch(draw.Count)) + { + _host.Draw(draw.Count, draw.InstanceCount, emission.FirstVertex, emission.FirstInstance); + break; + } + if (draw.Count != 3 || vertexInput.Buffers.Length != 0) { throw _host.Fatal($"A legacy rectangle list needs three vertices and no vertex buffers: count={draw.Count} buffers={vertexInput.Buffers.Length}."); diff --git a/src/SharpEmu.Libs/Gpu/Rendering/RenderExecutor.Targets.cs b/src/SharpEmu.Libs/Gpu/Rendering/RenderExecutor.Targets.cs index 38fa148..4c98f4b 100644 --- a/src/SharpEmu.Libs/Gpu/Rendering/RenderExecutor.Targets.cs +++ b/src/SharpEmu.Libs/Gpu/Rendering/RenderExecutor.Targets.cs @@ -24,30 +24,18 @@ public sealed partial class RenderExecutor return false; } - for (var slot = 0u; slot < ContextRegisters.ColorTargetCount; slot++) + for (var attempt = 0; attempt < 2; attempt++) { - if (slot != 0 && (context.RenderTargetMaskForSlot(slot) == 0 || context.ColorTargets[slot].BaseAddress == 0)) + ResolveTargetImages(context, ref state); + if (!TryDemoteMismatchedRenderScales(ref state)) { - continue; + break; } - if (ColorTargetResolver.Resolve(context, slot, DrawLayerOffset, ignoreTargetMask: false, out var resolvedSlot) is not { } resolution) - { - continue; - } - - var request = resolution.Request; - var image = _host.FindImage(ref request, exactFormat: false); - _host.BindRenderTarget(image); - state.Colors[(int)state.ColorCount++] = new ColorTargetState(in resolution, resolvedSlot, image); - } - - if (DepthTargetResolver.Resolve(context, _host.FormatSupport, _host.Fatal) is { } depthTarget) - { - var request = depthTarget.Target.Request; - var image = _host.FindImage(ref request, exactFormat: false); - _host.BindRenderTarget(image); - state.Depth = new DepthAttachmentState(in depthTarget, image); + // Every attachment is back at guest resolution now; resolve them again so the + // draw sees the replacements instead of the images that were just released. + state.ColorCount = 0; + state.Depth = default; } state.PixelActive = HasActivePixelShader(banks); @@ -62,6 +50,16 @@ public sealed partial class RenderExecutor return true; } + if (state.ColorCount == 0 && !state.Depth.HasTarget && !state.PixelActive && + PixelStageWritesMemory(banks, ref state)) + { + // A pixel shader can be used as a rasterized producer for a later + // visibility or culling pass. It has no framebuffer export, but its + // storage writes are still observable and must be executed. + TraceDrawDisposition(banks, in draw, "pixel-stores-only"); + return true; + } + if (state.ColorCount == 0 && !state.Depth.HasTarget && !state.PixelActive) { TraceDrawDisposition(banks, in draw, "no-framebuffer"); @@ -78,6 +76,73 @@ public sealed partial class RenderExecutor return true; } + private void ResolveTargetImages(ContextRegisters context, ref DrawState state) + { + for (var slot = 0u; slot < ContextRegisters.ColorTargetCount; slot++) + { + if (slot != 0 && (context.RenderTargetMaskForSlot(slot) == 0 || context.ColorTargets[slot].BaseAddress == 0)) + { + continue; + } + + if (ColorTargetResolver.Resolve(context, slot, DrawLayerOffset, ignoreTargetMask: false, out var resolvedSlot) is not { } resolution) + { + continue; + } + + var request = resolution.Request; + var image = _host.FindImage(ref request, exactFormat: false); + _host.BindRenderTarget(image); + state.Colors[(int)state.ColorCount++] = new ColorTargetState(in resolution, resolvedSlot, image, _host.GetRenderScale(image)); + } + + if (DepthTargetResolver.Resolve(context, _host.FormatSupport, _host.Fatal) is { } depthTarget) + { + var request = depthTarget.Target.Request; + var image = _host.FindImage(ref request, exactFormat: false); + _host.BindRenderTarget(image); + state.Depth = new DepthAttachmentState(in depthTarget, image, _host.GetRenderScale(image)); + } + } + + // One rendering scope has one render area, one viewport and one scissor, so its + // attachments have to agree on a host resolution. They disagree when one of them cannot + // be scaled at all, and then the whole pass goes back to guest resolution. + private bool TryDemoteMismatchedRenderScales(ref DrawState state) + { + if (!RenderScalePolicy.Enabled) + { + return false; + } + + var scale = AttachmentRenderScale(in state); + var mismatched = state.Depth.HasTarget && state.Depth.RenderScale != scale; + for (var i = 0; i < state.ColorCount && !mismatched; i++) + { + mismatched = state.Colors[i].RenderScale != scale; + } + + if (!mismatched) + { + return false; + } + + for (var i = 0; i < state.ColorCount; i++) + { + if (state.Colors[i].RenderScale != 1f) + { + _host.DemoteRenderScale(state.Colors[i].Image); + } + } + + if (state.Depth.HasTarget && state.Depth.RenderScale != 1f) + { + _host.DemoteRenderScale(state.Depth.Image); + } + + return true; + } + // Whether the bound vertex (or emulated geometry) stage stores to a buffer or a // storage image, which is the only observable effect a draw without attachments // and without a pixel stage can have. @@ -93,6 +158,25 @@ public sealed partial class RenderExecutor return stage.Program is { } program && (WritesStorageImage(program) || HasBufferWrites(stage)); } + private bool PixelStageWritesMemory(RegisterBanks banks, ref DrawState state) + { + if (banks.Shader.Pixel.Address == 0) + { + return false; + } + + state.PixelActive = true; + ResolveShaderPrograms(banks, ref state); + var stage = state.Programs.PixelInput.Stage; + if (!state.Programs.Available || stage.Program is not { } program) + { + state.PixelActive = false; + return false; + } + + return program.UsesDeviceAddresses || WritesStorageImage(program) || HasBufferWrites(stage); + } + // Color control mode 3 resolves slot 0 into slot 1 instead of drawing; true consumes the draw. private bool TryResolveMultisampleColor(ContextRegisters context) { @@ -175,7 +259,12 @@ public sealed partial class RenderExecutor readRequest.View.LevelCount, readRequest.View.BaseLayer, readRequest.View.LayerCount); - var extent = new Extent3D(write.Width, write.Height, 1); + // The copy moves host texels, so the guest extent crosses into the images' resolution. + var copyScale = _host.GetRenderScale(destination); + var extent = new Extent3D( + RenderScalePolicy.ScaleLength(write.Width, copyScale), + RenderScalePolicy.ScaleLength(write.Height, copyScale), + 1); var aspects = (depthCopy ? ImageAspectFlags.DepthBit : 0) | (stencilCopy ? ImageAspectFlags.StencilBit : 0); _host.MarkGpuWritten(destination); _host.CopyDepthStencilImage(source, destination, in range, in extent, aspects); @@ -192,7 +281,7 @@ public sealed partial class RenderExecutor var request = resolution.Request; var image = _host.FindImage(ref request, exactFormat: true); _host.BindRenderTarget(image); - return image.IsValid ? new ColorTargetState(in resolution, resolvedSlot, image) : null; + return image.IsValid ? new ColorTargetState(in resolution, resolvedSlot, image, _host.GetRenderScale(image)) : null; } private static bool IsSupportedSampleCount(uint samples) => samples is 1 or 2 or 4 or 8; @@ -231,7 +320,7 @@ public sealed partial class RenderExecutor ((pixelColorExportMasks >> (int)(slot * 4)) & 0xFu) == 0; // Acquires every attachment through the host and assembles the rendering scope. - private RenderingState AcquireAttachments(ref DrawState state) + private RenderingState AcquireAttachments(ref DrawState state, ContextRegisters context) { using var profileScope = RenderPhaseProfile.MeasureDetail(RenderPhaseProfile.Phase.DrawAttachmentPreparation); var rendering = new RenderingState @@ -269,8 +358,10 @@ public sealed partial class RenderExecutor } var view = target.Resolution.Request.View; - rendering.Width = Math.Min(rendering.Width, target.Resolution.Extent.Width); - rendering.Height = Math.Min(rendering.Height, target.Resolution.Extent.Height); + // The render area covers host texels, which a scaled attachment has more or fewer of. + var hostExtent = target.HostExtent; + rendering.Width = Math.Min(rendering.Width, hostExtent.Width); + rendering.Height = Math.Min(rendering.Height, hostExtent.Height); rendering.Layers = Math.Min(rendering.Layers, view.LayerCount); var clear = acquired.MetadataClear ? acquired.MetadataClearValue : target.Resolution.ColorClearValue; rendering.ColorAttachments[i] = new RenderingAttachment( @@ -306,17 +397,27 @@ public sealed partial class RenderExecutor } else if (attachmentSamples != target.Samples) { - throw _host.Fatal($"Mixed color and depth sample counts are not supported: color={attachmentSamples} depth={target.Samples}."); + if (_host.NativeTwoSampleMixedSupported && context.AntialiasingConfig.SampleCountLog2 == 1 && attachmentSamples == 1 && target.Samples == 2) + { + attachmentSamples = target.Samples; + } + else + throw _host.Fatal($"Mixed color and depth sample counts are not supported: color={attachmentSamples} depth={target.Samples}."); } var loadState = depth.LoadState; - var layout = _host.SamplesDepthAttachment(in depth) - ? loadState.AttachmentLayout(target.Format) - : DepthStencilState.WritableAttachmentLayout(target.Format); - _host.TransitionDepthAttachment(in depth, layout, loadState.AttachmentWriteAspects(target.Format)); + var layout = DepthStencilState.AttachmentLayoutFor( + loadState, + target.Format, + _host.SamplesDepthAttachment(in depth), + _host.DepthAttachmentLayout(in depth)); + if (_host.NativeTwoSampleMixedSupported && target.Samples == 2) + _host.ConfigureDepthSampleLocations(in depth, context.SampleLocations.Locations, layout, loadState.AttachmentWriteAspects(target.Format)); + else + _host.TransitionDepthAttachment(in depth, layout, loadState.AttachmentWriteAspects(target.Format)); var view = target.Request.View; - rendering.Width = Math.Min(rendering.Width, target.Width); - rendering.Height = Math.Min(rendering.Height, target.Height); + rendering.Width = Math.Min(rendering.Width, depth.HostWidth); + rendering.Height = Math.Min(rendering.Height, depth.HostHeight); rendering.Layers = Math.Min(rendering.Layers, view.LayerCount); var aspects = ViewFormatRules.DepthAspects(target.Format); rendering.DepthStencilAttachment = new RenderingAttachment( diff --git a/src/SharpEmu.Libs/Gpu/Rendering/RenderExecutor.Tessellation.cs b/src/SharpEmu.Libs/Gpu/Rendering/RenderExecutor.Tessellation.cs new file mode 100644 index 0000000..5d9fbd3 --- /dev/null +++ b/src/SharpEmu.Libs/Gpu/Rendering/RenderExecutor.Tessellation.cs @@ -0,0 +1,164 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Buffers.Binary; +using SharpEmu.HLE.GpuMemory; +using SharpEmu.Libs.Gpu.GpuCommands; +using SharpEmu.Libs.Gpu.GpuCommands.Registers; +using SharpEmu.Libs.Gpu.Pipelines; +using SharpEmu.Libs.Gpu.Scheduling; +using SharpEmu.ShaderCompiler; +using SharpEmu.ShaderCompiler.Vulkan; +using SharpEmu.Libs.VideoOut; +using Silk.NET.Vulkan; + +namespace SharpEmu.Libs.Gpu.Rendering; + +public sealed partial class RenderExecutor +{ + private void DrawTessellationIndexed(ulong submitId, RegisterBanks banks, in DrawIndexedArguments arguments) + { + var count = arguments.IndexCount; + var instances = arguments.InstanceCount; + var vertexOffset = unchecked(arguments.BaseVertex + + (arguments.OffsetSource == DrawOffsetSource.IndirectArguments ? 0 : (int)banks.UserConfig.IndexOffset)); + var firstInstance = arguments.FirstInstance; + var address = arguments.IndexAddress; + var size = arguments.IndexTypeAndSize switch + { + (uint)GuestIndexType.Index8 => 1u, (uint)GuestIndexType.Index16 => 2u, (uint)GuestIndexType.Index32 => 4u, + _ => throw _host.Fatal("The tessellation index format is unsupported."), + }; + if (arguments.IndirectArgumentsAddress != 0) + { + Span data = stackalloc byte[20]; + if (!_host.TryReadGuest(arguments.IndirectArgumentsAddress, data)) + throw _host.Fatal("The tessellation indirect draw arguments are unreadable."); + count = BinaryPrimitives.ReadUInt32LittleEndian(data); + instances = BinaryPrimitives.ReadUInt32LittleEndian(data[4..]); + var firstIndex = BinaryPrimitives.ReadUInt32LittleEndian(data[8..]); + if (!arguments.UnboundedIndexBuffer && (ulong)firstIndex + count > arguments.IndexCount) + throw _host.Fatal("The tessellation indirect draw exceeds the supplied index buffer."); + address = checked(address + (ulong)firstIndex * size); + vertexOffset = BinaryPrimitives.ReadInt32LittleEndian(data[12..]); + firstInstance = BinaryPrimitives.ReadUInt32LittleEndian(data[16..]); + } + DrawTessellation(submitId, banks, count, instances, address, size, vertexOffset, firstInstance); + } + + private void DrawTessellation(ulong submitId, RegisterBanks banks, uint count, uint instances, + ulong indexAddress, uint indexSize, int vertexOffset, uint firstInstance) + { + if (count == 0 || instances == 0) return; + ValidateDrawRegisters(banks); + var draw = new DrawCall("DrawTessellation", RecordedOperation.DrawIndexAuto, count, instances, firstInstance); + var state = DrawState.Create(); + if (!TryResolveDrawTargets(banks, draw, ref state)) { _host.ResetBindings(); return; } + ResolveShaderPrograms(banks, ref state); + var tessellation = state.Programs.Tessellation ?? throw _host.Fatal("The draw has no prepared tessellation programs."); + var hull = tessellation.Hull; + var input = hull.Input; + var layout = tessellation.HullConfiguration; + var patches = count / layout.InputControlPoints; + if (patches == 0) { _host.ResetBindings(); return; } + var factorIndex = input.Stage.Program!.TessellationFactorBuffer; + if (factorIndex < 0 || factorIndex >= input.Stage.Resources.Buffers.Length) + throw _host.Fatal("The tessellation factor buffer is missing from the hull resources."); + var descriptor = BufferDescriptorWords.From(input.Stage.Resources.Buffers[factorIndex]); + var factorSize = descriptor.Footprint() ?? throw _host.Fatal("The tessellation factor-ring footprint overflows."); + var patchFactorBytes = Gen5TessellationBridge.FactorCount(tessellation.Configuration.Domain) * 4; + var groupFactorBytes = layout.PatchesPerGroup * patchFactorBytes; + var factorBytes = Math.Min(patches, layout.PatchesPerGroup) * patchFactorBytes; + if (descriptor.Address == 0 || factorSize < factorBytes || descriptor.SwizzleEnabled || descriptor.AddThreadId) + throw _host.Fatal("The tessellation factor ring has an invalid address, extent or addressing mode."); + // Groups that run together take their own off-chip buffer and their own run of the + // factor ring. Without the guest's off-chip configuration, one group runs at a time. + var (offchipBuffers, offchipSlotBytes) = Gpu.TessellationOffchip.Configuration; + var groupsPerBatch = offchipBuffers == 0 ? 1u : (uint)Math.Clamp(Math.Min(offchipBuffers, factorSize / groupFactorBytes), 1, uint.MaxValue); + var batchCapacity = groupsPerBatch * layout.PatchesPerGroup; + var originalHullStage = input.Stage; + var originalDomainStage = state.Programs.VertexInput.Stage; + var pipeline = _pipelines.CreateComputePipeline(input, hull.Program); + // Patches are numbered across all instances, so instances share batches; each shader + // thread splits its patch number into an instance and a patch within it. + var totalPatches = (ulong)instances * patches; + if (totalPatches > uint.MaxValue) + throw _host.Fatal("The tessellated draw has more patches than one batch numbering covers."); + if (RenderPhaseProfile.Enabled) + { + TessellationProfile.RecordDraw(instances, patches, (long)((totalPatches + batchCapacity - 1) / batchCapacity), groupsPerBatch == 1); + } + + Span ranges = stackalloc GuestSpan[2]; + try + { + for (uint firstPatch = 0; firstPatch < (uint)totalPatches;) + { + var batchPatches = Math.Min(batchCapacity, (uint)totalPatches - firstPatch); + var groups = (batchPatches + layout.PatchesPerGroup - 1) / layout.PatchesPerGroup; + var data = new uint[Gen5TessellationData.DwordCount]; + data[Gen5TessellationData.FirstPatch] = firstPatch; + data[Gen5TessellationData.PatchCount] = batchPatches; + data[Gen5TessellationData.VertexOffset] = unchecked((uint)vertexOffset); + data[Gen5TessellationData.InstanceId] = firstInstance; + data[Gen5TessellationData.PatchesPerInstance] = patches; + data[Gen5TessellationData.IndexSize] = indexSize; + data[Gen5TessellationData.FactorBytes] = batchPatches * patchFactorBytes; + data[Gen5TessellationData.GroupPatches] = offchipBuffers == 0 ? 0 : layout.PatchesPerGroup; + data[Gen5TessellationData.OffchipSlotBytes] = offchipSlotBytes; + data[Gen5TessellationData.FactorGroupBytes] = groupFactorBytes; + data[Gen5TessellationData.MinimumLevel] = banks.Context.ShaderInterface.MinTessellationLevel; + data[Gen5TessellationData.MaximumLevel] = banks.Context.ShaderInterface.MaxTessellationLevel; + using (_host.BeginPreparation()) + { + _host.EndRendering(); + input.Stage = originalHullStage with { TessellationData = data }; + var bindings = _host.PrepareBindings(input.Stage); + ranges[0] = new(descriptor.Address, factorSize); + var rangeCount = 1; + if (indexSize != 0) + { + if (indexAddress == 0) throw _host.Fatal("The tessellation indexed draw has no index buffer."); + ranges[rangeCount++] = new(indexAddress, (ulong)count * indexSize); + } + _host.PrepareBufferAllocations(ranges[..rangeCount]); + if (input.Stage.Program!.UsesDeviceAddresses) _host.PrepareDeviceAddresses(); + void Address(uint word, ulong address) + { + data[word] = (uint)address; data[word + 1] = (uint)(address >> 32); + } + Address(Gen5TessellationData.FactorAddress, _host.ObtainBufferDeviceAddress(descriptor.Address, factorSize, true)); + if (indexSize != 0) + { + var bias = (uint)(indexAddress & 3); + var bytes = ((ulong)count * indexSize + bias + 3) & ~3ul; + Address(Gen5TessellationData.IndexAddress, _host.ObtainBufferDeviceAddress(indexAddress - bias, bytes, false)); + data[Gen5TessellationData.IndexByteOffset] = bias; + } + _host.UpdateTessellationData(bindings, data); + // Binding uploads shader data. Populate physical pointers + // before that upload, rather than only before descriptor commit. + _host.BindResources(bindings); + _host.CommitBindings(PipelineBindPoint.Compute, pipeline, [bindings]); + _host.BindPipeline(PipelineBindPoint.Compute, pipeline); + _host.Dispatch(groups, 1, 1); + _host.ShaderWriteBarrier(PipelineStageFlags.ComputeShaderBit); + } + state.Programs.VertexInput.Stage = originalDomainStage with { TessellationData = data }; + var batch = draw with { Count = batchPatches * layout.InputControlPoints, InstanceCount = 1 }; + var emission = new DrawEmission(false, 0, 0, 0); + RecordDraw(submitId, banks, batch, ref state, PrimitiveTopology.PatchList, emission, default, false, false, false); + _host.EndRendering(); + // The next batch reuses relative patch IDs in the guest rings. + _host.ShaderWriteHazardBarrier(); + firstPatch += batchPatches; + } + } + finally + { + input.Stage = originalHullStage; + state.Programs.VertexInput.Stage = originalDomainStage; + _host.ResetBindings(); + } + } +} diff --git a/src/SharpEmu.Libs/Gpu/Rendering/RenderExecutor.cs b/src/SharpEmu.Libs/Gpu/Rendering/RenderExecutor.cs index 8558cf4..46793c5 100644 --- a/src/SharpEmu.Libs/Gpu/Rendering/RenderExecutor.cs +++ b/src/SharpEmu.Libs/Gpu/Rendering/RenderExecutor.cs @@ -7,6 +7,7 @@ using SharpEmu.Libs.Gpu.GpuCommands.Registers; using SharpEmu.Libs.Gpu.Images; using SharpEmu.Libs.Gpu.Scheduling; using SharpEmu.Libs.VideoOut; +using SharpEmu.ShaderCompiler; using Silk.NET.Vulkan; using ResourceSnapshot = SharpEmu.ShaderCompiler.Resources.ResourceSnapshot; @@ -49,6 +50,7 @@ public sealed partial class RenderExecutor (stages & ~VgtShaderStagesWaveSizeBits) is 0x02002000 or 0x00002000 or 0x00002030; private const uint MaxOutputPerSubgroupLimit = 0x40; private static int _geometryWarningShown; + private static int _tessellationWarningShown; private readonly IRenderHost _host; private readonly IShaderPipelineProvider _pipelines; @@ -208,6 +210,15 @@ public sealed partial class RenderExecutor return; } + if (Pipelines.ShaderPipelineCache.IsMergedTessellationMask(banks.Context.ShaderStages)) + { + if (IsSupportedTessellation(banks)) + { + DrawTessellationIndexed(submitId, banks, arguments); + } + + return; + } if (!HasValidVertexShader(shader) || IsUnsupportedGeometryStage(banks)) { return; @@ -309,6 +320,50 @@ public sealed partial class RenderExecutor } public void DrawAuto(ulong submitId, RegisterBanks banks, in DrawAutoArguments arguments) + { + if (arguments.IndirectArgumentsAddress != 0 && !CanDrawAutoIndirectOnGpu(banks)) + { + DrawAutoWithCpuArguments(submitId, banks, in arguments); + return; + } + + DrawAutoCore(submitId, banks, in arguments); + } + + // Fans, polygons and legacy rectangle/quad lists are drawn from their vertex counts, so + // those read the indirect arguments on the CPU, as merged tessellation does: it counts its + // patches from the vertex count. Rectangle lists draw through the host's general + // rectangle-list variant, as indexed indirect draws do. + private static bool CanDrawAutoIndirectOnGpu(RegisterBanks banks) => + !Pipelines.ShaderPipelineCache.IsMergedTessellationMask(banks.Context.ShaderStages) && + (GuestPrimitiveType)banks.UserConfig.PrimitiveType is + GuestPrimitiveType.PointList or GuestPrimitiveType.LineList or GuestPrimitiveType.LineStrip or + GuestPrimitiveType.TriangleList or GuestPrimitiveType.TriangleStrip or GuestPrimitiveType.RectangleList; + + private void DrawAutoWithCpuArguments(ulong submitId, RegisterBanks banks, in DrawAutoArguments arguments) + { + + Span bytes = stackalloc byte[(int)AutoIndirectArgumentsSize]; + if (!_host.TryReadGuest(arguments.IndirectArgumentsAddress, bytes)) + { + throw _host.Fatal($"The indirect draw arguments are unreadable: address=0x{arguments.IndirectArgumentsAddress:X16}."); + } + + var words = System.Runtime.InteropServices.MemoryMarshal.Cast(bytes); + var resolved = arguments with + { + VertexCount = words[0], + InstanceCount = words[1], + FirstVertex = words[2], + FirstInstance = words[3], + IndirectArgumentsAddress = 0, + }; + DrawAutoCore(submitId, banks, in resolved); + } + + private const ulong AutoIndirectArgumentsSize = 16; + + private void DrawAutoCore(ulong submitId, RegisterBanks banks, in DrawAutoArguments arguments) { using var profileScope = RenderPhaseProfile.MeasureDetail(RenderPhaseProfile.Phase.DrawExecutor); if (!_host.IsRecording) @@ -336,6 +391,17 @@ public sealed partial class RenderExecutor return; } + if (Pipelines.ShaderPipelineCache.IsMergedTessellationMask(banks.Context.ShaderStages)) + { + var tessellationVertexOffset = unchecked((int)arguments.FirstVertex + + (arguments.OffsetSource == DrawOffsetSource.IndirectArguments ? 0 : (int)banks.UserConfig.IndexOffset)); + if (IsSupportedTessellation(banks)) + { + DrawTessellation(submitId, banks, arguments.VertexCount, arguments.InstanceCount, 0, 0, tessellationVertexOffset, arguments.FirstInstance); + } + + return; + } if (!HasValidVertexShader(shader) || IsUnsupportedGeometryStage(banks)) { return; @@ -357,13 +423,20 @@ public sealed partial class RenderExecutor return; } - if (!ResolveTopology(userConfig, autoDraw: true, out var topology)) + if (!ResolveTopology(userConfig, autoDraw: true, out var topology, arguments.VertexCount)) { TraceDrawDisposition(banks, in draw, "no-primitive-topology"); _host.ResetBindings(); return; } + if (arguments.IndirectArgumentsAddress != 0 && state.ColorCount == 0 && !state.Depth.HasTarget) + { + // A targetless draw may be retained and replayed later; it needs its counts. + DrawAutoWithCpuArguments(submitId, banks, in arguments); + return; + } + ResolveShaderPrograms(banks, ref state); var vertexInput = state.Programs.VertexInput; var pixelInput = state.Programs.PixelInput; @@ -396,7 +469,8 @@ public sealed partial class RenderExecutor false, 0, (uint)vertexOffset, - indirect ? arguments.FirstInstance : ResolveInstanceOffset(vertexInput)); + indirect ? arguments.FirstInstance : ResolveInstanceOffset(vertexInput), + arguments.IndirectArgumentsAddress); RecordDraw(submitId, banks, in draw, ref state, topology, in emission, default, primitiveRestart: false, setBindDebug: false, setAutoDebug: true); _host.ResetBindings(); } @@ -496,7 +570,32 @@ public sealed partial class RenderExecutor private static bool IsKnownGeometryOutputPrimitiveType(uint value) => value <= 4; // Only the plain vertex path and the primitive-shader vertex path with default geometry state run. - private static bool IsUnsupportedGeometryStage(RegisterBanks banks) + // A tessellation configuration the bridge cannot express skips the draw, as an + // unsupported geometry stage does, instead of stopping the emulator. + private static bool IsSupportedTessellation(RegisterBanks banks) + { + var shaderInterface = banks.Context.ShaderInterface; + string? error = null; + if (!Gen5TessellationInfo.TryDecode(shaderInterface.TessellationFactorParameter, out var tessellation, out var factorError)) + error = factorError; + else if (tessellation.Spacing == Gen5TessellationSpacing.PowerOfTwo) + error = "power-of-two partitioning needs factor conversion"; + else if (!Gen5TessellationHullInfo.TryDecode(shaderInterface.LocalHullConfiguration, 0, out _, out var hullError)) + error = hullError; + if (error is null) + return true; + + if (Interlocked.Exchange(ref _tessellationWarningShown, 1) == 0) + { + Console.Error.WriteLine( + $"Warning: the title uses an unsupported tessellation configuration; those draw calls are skipped. {error} " + + $"tf_param=0x{shaderInterface.TessellationFactorParameter:X8} ls_hs_config=0x{shaderInterface.LocalHullConfiguration:X8}"); + } + + return false; + } + + private bool IsUnsupportedGeometryStage(RegisterBanks banks) { var context = banks.Context; var shaderInterface = context.ShaderInterface; @@ -556,7 +655,8 @@ public sealed partial class RenderExecutor { var context = banks.Context; var shaderInterface = context.ShaderInterface; - var hasColorOutput = (context.RenderTargetMask & shaderInterface.ColorShaderMask) != 0; + var hasColorOutput = context.ColorControl.Mode != 0 && + (context.RenderTargetMask & shaderInterface.ColorShaderMask) != 0; return banks.Shader.Pixel.Address != 0 && (hasColorOutput || PixelShaderHasDepthOrCoverageSideEffects(shaderInterface)); } @@ -572,7 +672,9 @@ public sealed partial class RenderExecutor return mode is ColorModeEliminateFastClear or ColorModeFmaskDecompress or ColorModeDccDecompress; } - public bool ResolveTopology(UserConfigRegisters userConfig, bool autoDraw, out PrimitiveTopology topology) + private static bool IsLegacyRectangleBatch(uint vertexCount) => vertexCount > 3 && vertexCount % 3 == 0; + + public bool ResolveTopology(UserConfigRegisters userConfig, bool autoDraw, out PrimitiveTopology topology, uint vertexCount = 3) { topology = PrimitiveTopology.PointList; switch ((GuestPrimitiveType)userConfig.PrimitiveType) @@ -607,7 +709,9 @@ public sealed partial class RenderExecutor throw _host.Fatal($"The primitive type is unknown for an indexed draw: primitiveType={userConfig.PrimitiveType}."); } - topology = PrimitiveTopology.TriangleStrip; + // Several rectangles take the rectangle-list path, which derives each + // fourth corner from three vertices like the hardware. + topology = IsLegacyRectangleBatch(vertexCount) ? PrimitiveTopology.PatchList : PrimitiveTopology.TriangleStrip; break; case GuestPrimitiveType.QuadListLegacy: topology = PrimitiveTopology.TriangleFan; diff --git a/src/SharpEmu.Libs/Gpu/Rendering/ShaderStageResources.cs b/src/SharpEmu.Libs/Gpu/Rendering/ShaderStageResources.cs index fa55ad3..8c535e7 100644 --- a/src/SharpEmu.Libs/Gpu/Rendering/ShaderStageResources.cs +++ b/src/SharpEmu.Libs/Gpu/Rendering/ShaderStageResources.cs @@ -4,6 +4,7 @@ using SharpEmu.Libs.Gpu.Images; using SharpEmu.Libs.Gpu.Scheduling; using SharpEmu.ShaderCompiler.Resources; +using SharpEmu.ShaderCompiler; using ResourceSnapshot = SharpEmu.ShaderCompiler.Resources.ResourceSnapshot; namespace SharpEmu.Libs.Gpu.Rendering; @@ -14,6 +15,7 @@ public enum ShaderStageKind Vertex, Pixel, Compute, + TessellationEvaluation, } public enum ImageResourceClass : byte @@ -42,6 +44,7 @@ public class ShaderProgramInfo public const int NoScalarRegister = -1; public ShaderStageKind Stage { get; init; } + public int TessellationFactorBuffer { get; init; } = -1; public ulong Hash { get; init; } public uint UserDataBase { get; init; } public uint UserDataCount { get; init; } @@ -55,6 +58,7 @@ public class ShaderProgramInfo // The dword every invocation stores when the whole program is 'index, v_mov constant, one // buffer store, end' (AGC's constant fill kernels); null for any other program. public uint? ConstantStoreValue { get; init; } + public Pipelines.ImmediateConstantFill? ImmediateConstantFill { get; init; } public bool HasBitwiseExclusiveOr { get; init; } public Pipelines.ConstantFill? ConstantFill { get; init; } public Pipelines.BoundedFill? BoundedFill { get; init; } @@ -79,6 +83,19 @@ public readonly record struct ShaderStageResources(ShaderProgramInfo? Program, R public bool IsValid => Program is not null; public DispatchThreadLimits? ThreadLimits { get; init; } + public uint[]? TessellationData { get; init; } + + public void WriteTessellationData(Span shaderData) + { + if (Program?.Bindings is not { UsesTessellationData: true } layout) return; + if (TessellationData is not { Length: (int)Gen5TessellationData.DwordCount } data || + shaderData.Length != layout.ShaderDataDwordCount) + throw SubmissionScheduler.Fatal("The tessellation stage has missing runtime data or invalid shader data."); + data.CopyTo(shaderData[(int)layout.TessellationDataDword..]); + } + + // The host resolution of the attachments this draw writes; null outside a draw. + public float? AttachmentRenderScale { get; init; } public void WriteDispatchThreadLimits(Span shaderData) { @@ -93,6 +110,25 @@ public readonly record struct ShaderStageResources(ShaderProgramInfo? Program, R shaderData[offset + 1] = limits.Y; shaderData[offset + 2] = limits.Z; } + + // The internal-resolution dwords: the factor that maps one guest pixel onto host texels, + // its reciprocal, which image resources the host holds at that factor, and the factor the + // pixel position is divided by so the program keeps reading guest pixels. + public void WriteRenderScale(Span shaderData, ulong scaledImages, float imageScale) + { + if (Program?.Bindings is not { UsesRenderScale: true } layout) return; + if (shaderData.Length != layout.ShaderDataDwordCount || !float.IsFinite(imageScale) || imageScale <= 0) + { + throw SubmissionScheduler.Fatal("The draw has invalid shader data or an invalid scale for its render scale dwords."); + } + + var offset = (int)layout.RenderScaleDword; + shaderData[offset] = BitConverter.SingleToUInt32Bits(imageScale); + shaderData[offset + 1] = BitConverter.SingleToUInt32Bits(1.0f / imageScale); + shaderData[offset + 2] = (uint)scaledImages; + shaderData[offset + 3] = (uint)(scaledImages >> 32); + shaderData[offset + 4] = BitConverter.SingleToUInt32Bits(1.0f / (AttachmentRenderScale ?? 1.0f)); + } } public readonly record struct DispatchThreadLimits(uint X, uint Y, uint Z); @@ -100,7 +136,9 @@ public readonly record struct DispatchThreadLimits(uint X, uint Y, uint Z); // The vertex buffer words of one fetch slot as the vertex program declares them. public readonly record struct VertexInputBuffer(ulong Address, uint Stride, uint RecordCount, bool PerInstance = false) { - public ulong Size => Stride != 0 ? (ulong)Stride * RecordCount : RecordCount; + public uint MinimumFetchBytes { get; init; } + + public ulong Size => Math.Max(Stride != 0 ? (ulong)Stride * RecordCount : RecordCount, MinimumFetchBytes); } // One attribute of the vertex tables: its buffer words, the registers it fills and its buffer slot. @@ -125,6 +163,7 @@ public readonly record struct ClipSpaceTransform( public sealed class VertexInputInfo { + public Pipelines.TessellationPipelineStages? Tessellation { get; init; } public const int MaxBuffers = 32; public VertexInputBuffer[] Buffers { get; init; } = []; @@ -150,6 +189,7 @@ public sealed class PixelInputInfo public uint PerspectiveCenterRegister { get; init; } = NoPerspectiveCenterRegister; public uint[] InterpolatorSettings { get; init; } = new uint[InterpolatorCount]; public byte[] TargetOutputModes { get; init; } = new byte[TargetCount]; + public byte[] TargetExportFormats { get; init; } = new byte[TargetCount]; public ColorComponentMap[] TargetExportMappings { get; init; } = new ColorComponentMap[TargetCount]; public uint ScratchDwords { get; init; } public bool PositionX { get; init; } @@ -161,8 +201,13 @@ public sealed class PixelInputInfo public bool KillEnable { get; init; } public bool DepthExportEnable { get; init; } public bool SampleMaskExportEnable { get; init; } + public uint MaskExportSamples { get; init; } = 1; + public uint RasterizationSamples { get; init; } = 1; + public uint? InterpolationSample { get; init; } + public IReadOnlyList<(float X, float Y)> CustomSampleOffsets { get; init; } = []; public bool SampleShading { get; init; } public bool EarlyDepth { get; init; } + public uint ShaderSampleExclusionMask { get; init; } public bool ExecuteOnNoop { get; init; } public ShaderStageResources Stage { get; set; } diff --git a/src/SharpEmu.Libs/Gpu/Rendering/TessellationProfile.cs b/src/SharpEmu.Libs/Gpu/Rendering/TessellationProfile.cs new file mode 100644 index 0000000..aa0c279 --- /dev/null +++ b/src/SharpEmu.Libs/Gpu/Rendering/TessellationProfile.cs @@ -0,0 +1,44 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +namespace SharpEmu.Libs.Gpu.Rendering; + +// Counts how tessellated draws are split into hull dispatches and domain draws; part of the +// render profile. Each batch prepares and binds every stage again, so batches per draw is +// the multiplier on the tessellation cost. +internal static class TessellationProfile +{ + private static long _draws; + private static long _instances; + private static long _patches; + private static long _batches; + private static long _singleGroupBatches; + private static long _maxBatchesPerDraw; + + public static void RecordDraw(uint instances, uint patches, long batches, bool oneGroupPerBatch) + { + _draws++; + _instances += instances; + _patches += (long)patches * instances; + _batches += batches; + if (oneGroupPerBatch) + { + _singleGroupBatches += batches; + } + + _maxBatchesPerDraw = Math.Max(_maxBatchesPerDraw, batches); + } + + public static string? TakeReport() + { + if (_draws == 0) + { + return null; + } + + var report = string.Create(System.Globalization.CultureInfo.InvariantCulture, + $"[PERF][TESSELLATION] draws={_draws} instances={_instances} patches={_patches} batches={_batches} batches_per_draw={(double)_batches / _draws:F1} max_batches_per_draw={_maxBatchesPerDraw} single_group_batches={_singleGroupBatches} offchip_buffers={TessellationOffchip.Configuration.Buffers}"); + _draws = _instances = _patches = _batches = _singleGroupBatches = _maxBatchesPerDraw = 0; + return report; + } +} diff --git a/src/SharpEmu.Libs/Gpu/Scheduling/SubmissionScheduler.cs b/src/SharpEmu.Libs/Gpu/Scheduling/SubmissionScheduler.cs index 7999a4d..f9df365 100644 --- a/src/SharpEmu.Libs/Gpu/Scheduling/SubmissionScheduler.cs +++ b/src/SharpEmu.Libs/Gpu/Scheduling/SubmissionScheduler.cs @@ -86,6 +86,9 @@ public sealed class SubmissionScheduler : IGpuTickScheduler, IDisposable public TickTimeline Timeline => _timeline; + // Commands recorded into the open command buffer that the GPU has not received yet. + internal bool HasUnsubmittedCommands => !_command.IsInvalid && _command.HasPendingCommands; + internal bool PriorityWorkerAlive => _priorityThread.IsAlive; // Report the fatal error, then return an exception to stop the caller. @@ -272,6 +275,18 @@ public sealed class SubmissionScheduler : IGpuTickScheduler, IDisposable _lastMemoryDrainTick = nextTick; } + // Waits for a tick that has already been submitted; safe from any thread, since it only + // waits on the timeline and records nothing. + public void WaitForSubmittedTick(ulong tick) + { + if (tick >= CurrentTick) + { + throw Fatal($"Cannot wait off the queue for a tick that is still recording: {tick} >= {CurrentTick}."); + } + + _timeline.Wait(tick); + } + public void Wait(ulong tick) { if (tick > CurrentTick) @@ -297,9 +312,53 @@ public sealed class SubmissionScheduler : IGpuTickScheduler, IDisposable } } - public void RunCompletedOperations() + // Reading the timeline semaphore is a driver call; the per-draw housekeeping asked for it + // twice per draw, about 12 % of the command thread in Ghost of Yotei. That path polls at + // most once per interval and otherwise uses the last value: completion work then runs a + // fraction of a millisecond later, and every real wait still reads the timeline at once. + private static readonly long DefaultTimelinePollInterval = System.Diagnostics.Stopwatch.Frequency * + (long.TryParse(Environment.GetEnvironmentVariable("SHARPEMU_TIMELINE_POLL_US"), out var pollUs) ? pollUs : 500) / 1_000_000; + private long _lastTimelinePoll; + + // Stopwatch ticks between two polled timeline reads; zero reads it on every poll. + internal long TimelinePollInterval { get; set; } = DefaultTimelinePollInterval; + + private void PollTimeline() { + var now = System.Diagnostics.Stopwatch.GetTimestamp(); + if (now - _lastTimelinePoll < TimelinePollInterval) + { + return; + } + + _lastTimelinePoll = now; _timeline.RefreshCompletedTick(); + } + + // IsTickComplete for the per-draw housekeeping: the timeline is read at most once per poll interval. + public bool IsTickCompletePolled(ulong tick) + { + if (!_timeline.IsTickComplete(tick)) + { + PollTimeline(); + } + + return _timeline.IsTickComplete(tick); + } + + public void RunCompletedOperations() => RunCompletedOperations(polled: false); + + public void RunCompletedOperations(bool polled) + { + if (polled) + { + PollTimeline(); + } + else + { + _timeline.RefreshCompletedTick(); + } + for (;;) { TickWork operation; diff --git a/src/SharpEmu.Libs/Gpu/Scheduling/TickOwnedResources.cs b/src/SharpEmu.Libs/Gpu/Scheduling/TickOwnedResources.cs new file mode 100644 index 0000000..9018f3a --- /dev/null +++ b/src/SharpEmu.Libs/Gpu/Scheduling/TickOwnedResources.cs @@ -0,0 +1,45 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +namespace SharpEmu.Libs.Gpu.Scheduling; + +// A GPU feedback resource belongs to one submission until its completion callback +// has consumed it. Later submissions must never reuse its mapped storage. +internal sealed class TickOwnedResources : IDisposable where T : class, IDisposable +{ + private readonly Dictionary _resources = []; + private readonly object _gate = new(); + + public T Acquire(ulong tick, Func create) + { + lock (_gate) + { + if (!_resources.TryGetValue(tick, out var resource)) + { + resource = create(); + _resources.Add(tick, resource); + } + return resource; + } + } + + public void Complete(ulong tick, Action consume) + { + lock (_gate) + { + if (!_resources.Remove(tick, out var resource)) return; + try { consume(resource); } + finally { resource.Dispose(); } + } + } + + // The caller must wait for GPU completion before disposing pending resources. + public void Dispose() + { + lock (_gate) + { + foreach (var resource in _resources.Values) resource.Dispose(); + _resources.Clear(); + } + } +} diff --git a/src/SharpEmu.Libs/Gpu/TessellationOffchip.cs b/src/SharpEmu.Libs/Gpu/TessellationOffchip.cs new file mode 100644 index 0000000..292d344 --- /dev/null +++ b/src/SharpEmu.Libs/Gpu/TessellationOffchip.cs @@ -0,0 +1,36 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +namespace SharpEmu.Libs.Gpu; + +// The hull-shader off-chip configuration the guest driver programs with +// sceAgcDriverSetHsOffchipParam (VGT_HS_OFFCHIP_PARAM). Each hull threadgroup in flight +// owns one buffer, so this bounds how many groups can run together and how far apart +// their off-chip data lies. +public static class TessellationOffchip +{ + private static long _configuration; + + // Zero buffers means the guest never configured off-chip memory. + public static (uint Buffers, uint SlotBytes) Configuration + { + get + { + var value = Volatile.Read(ref _configuration); + return ((uint)value, (uint)(value >> 32)); + } + } + + // OFFCHIP_BUFFERING [9:0] holds the buffer count minus one; OFFCHIP_GRANULARITY [11:10] + // selects 8K, 16K, 32K or 64K dwords per buffer. + public static (uint Buffers, uint SlotBytes) Decode(uint parameter) => + ((parameter & 0x3FF) + 1, (8u * 1024 * sizeof(uint)) << (int)((parameter >> 10) & 3)); + + public static void Configure(uint parameter) + { + var (buffers, slotBytes) = Decode(parameter); + Volatile.Write(ref _configuration, (long)((ulong)slotBytes << 32 | buffers)); + } + + internal static void Reset() => Volatile.Write(ref _configuration, 0); +} diff --git a/src/SharpEmu.Libs/Gpu/Vulkan/NativeHalfConversionProbe.cs b/src/SharpEmu.Libs/Gpu/Vulkan/NativeHalfConversionProbe.cs new file mode 100644 index 0000000..c8dbb1e --- /dev/null +++ b/src/SharpEmu.Libs/Gpu/Vulkan/NativeHalfConversionProbe.cs @@ -0,0 +1,392 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Numerics; +using System.Runtime.InteropServices; +using SharpEmu.Libs.Gpu.Buffers; +using SharpEmu.Libs.Gpu.Scheduling; +using SharpEmu.ShaderCompiler.Vulkan; +using Silk.NET.Vulkan; + +namespace SharpEmu.Libs.Gpu.Vulkan; + +// The CPU mirror of the two integer f16 conversion sequences the Gen5 translator emits by +// default, plus the vectors the device probe measures the native GLSL ext instructions with. +// Kept next to the probe because it is the reference the probe compares against: native is +// only used when it reproduces these results bit for bit. +internal static class HalfConversionReference +{ + // Mirrors Gen5SpirvTranslator.EmitHalfToFloat: the f32 bits of the f16 in the low 16 bits. + public static uint HalfToFloatBits(uint halfBits) + { + var sign = (halfBits & 0x8000u) << 16; + var exponent = (halfBits >> 10) & 0x1Fu; + var mantissa = halfBits & 0x3FFu; + + var normal = ((exponent + 112u) << 23) | (mantissa << 13); + var infinityNan = 0x7F80_0000u | (mantissa << 13); + + var highBit = (uint)BitOperations.Log2(mantissa | 1u); + var shift = (int)(23u - highBit); + var subFraction = (mantissa << shift) & 0x7F_FFFFu; + var subnormal = mantissa != 0 ? (((highBit + 103u) << 23) | subFraction) : 0u; + + var magnitude = exponent == 0 ? subnormal : exponent == 31 ? infinityNan : normal; + return sign | magnitude; + } + + // Mirrors Gen5SpirvTranslator.EmitFloatToHalf: the f16 bits, round to nearest even. + // NaN collapses to one quiet payload (sign | 0x7C00 | 0x200) by construction. + public static uint FloatToHalfBits(uint bits) + { + var sign = (bits >> 16) & 0x8000u; + var absolute = bits & 0x7FFF_FFFFu; + + var isInfinityNan = absolute >= 0x7F80_0000u; + var isNan = absolute > 0x7F80_0000u; + var infinityNan = sign | 0x7C00u | (isNan ? 0x200u : 0u); + + var exponent = absolute >> 23; + var mantissa = absolute & 0x7F_FFFFu; + var significand = mantissa | 0x80_0000u; + + var roundBit = (significand >> 13) & 1u; + var rounded = (significand + 0xFFFu + roundBit) >> 13; + var halfExponent = exponent - 112u; + var normalBits = (halfExponent << 10) + (rounded - 0x400u); + var normal = exponent >= 113u ? normalBits >= 0x7C00u ? 0x7C00u : normalBits : 0u; + + var distance = 126u - exponent; + var shift = (int)(distance > 25u ? 25u : distance); + var shiftMask = (1u << shift) - 1u; + var halfWay = 1u << (shift - 1); + var lowBits = significand & shiftMask; + var quotient = significand >> shift; + var roundUp = lowBits > halfWay || (lowBits == halfWay && (quotient & 1u) != 0u); + var subnormal = quotient + (roundUp ? 1u : 0u); + + var finite = exponent <= 112u ? subnormal : normal; + return isInfinityNan ? infinityNan : sign | finite; + } + + // Every f16 bit pattern, every f16 value widened to f32, the midpoint between adjacent + // f16 values and that midpoint one f32 ulp either way (the rounding decisions), the + // overflow boundary, signed zeroes, infinities, NaNs and f32 subnormals. Padded to a + // multiple of 64 so the probe dispatch needs no bounds check. Each entry is measured + // twice: as the f16 in its low 16 bits, and as an f32 bit pattern. + public static uint[] BuildTestVectors() + { + var values = new List(1 << 19); + for (uint bits = 0; bits <= 0xFFFFu; bits++) + { + values.Add(bits); + values.Add(HalfToFloatBits(bits)); + } + + // 0x7BFF is the largest finite f16; pair it with its successor to cover the overflow edge. + for (uint bits = 0; bits <= 0x7BFFu; bits++) + { + var low = (double)BitConverter.UInt32BitsToSingle(HalfToFloatBits(bits)); + var high = (double)BitConverter.UInt32BitsToSingle(HalfToFloatBits(bits + 1)); + var midpoint = BitConverter.SingleToUInt32Bits((float)(low + (high - low) / 2)); + for (var offset = -1; offset <= 1; offset++) + { + var candidate = (uint)(midpoint + offset); + values.Add(candidate); + values.Add(candidate | 0x8000_0000u); + } + } + + float[] specials = + [ + 0f, -0f, 1f, -1f, 65504f, -65504f, 65520f, -65520f, 65536f, -65536f, + 1e30f, -1e30f, float.MaxValue, float.MinValue, float.Epsilon, -float.Epsilon, + float.PositiveInfinity, float.NegativeInfinity, + 5.96046448e-8f, 6.09755516e-5f, -6.09755516e-5f, 3.05175781e-5f, 2.98023224e-8f, + ]; + foreach (var value in specials) + { + values.Add(BitConverter.SingleToUInt32Bits(value)); + } + + uint[] rawSpecials = + [ + 0x7F80_0001u, 0xFF80_0001u, 0x7FC0_0000u, 0xFFC0_0000u, 0x7FFF_FFFFu, 0xFFFF_FFFFu, + 0x0000_0001u, 0x8000_0001u, 0x007F_FFFFu, 0x807F_FFFFu, 0x0080_0000u, + ]; + values.AddRange(rawSpecials); + + while (values.Count % 64 != 0) + { + values.Add(0); + } + + return [.. values]; + } + + // The probe's match rule. Every finite value, every zero and every infinity must be + // bit-exact, sign included. A NaN only has to stay a NaN: the payload and the sign of a + // NaN are not defined by either side. The emulation picks its own payload for f32->f16 + // (sign | 0x7C00 | 0x200) and carries the f16 payload through for f16->f32, while the + // measured hardware convert canonicalizes to a positive quiet NaN and drops the sign + // (Apple M4: every one of the 2314 disagreements out of 643200 is a negative NaN whose + // sign the hardware cleared). Nothing a shader computes from a NaN can see that + // difference short of inspecting the bits, and the alternative is paying ~20 integer + // ops on every f16 conversion in the program to define a NaN payload the ISA leaves open. + public static bool Matches(uint expected, uint actual, bool asHalf) + { + if (expected == actual) + { + return true; + } + + return asHalf + ? IsHalfNan(expected) && IsHalfNan(actual) + : IsFloatNan(expected) && IsFloatNan(actual); + } + + private static bool IsHalfNan(uint bits) => (bits & 0x7C00u) == 0x7C00u && (bits & 0x3FFu) != 0; + + private static bool IsFloatNan(uint bits) => (bits & 0x7F80_0000u) == 0x7F80_0000u && (bits & 0x7F_FFFFu) != 0; +} + +// Runs the fixed f16 conversion probe once per device: the native GLSL UnpackHalf2x16 / +// PackHalf2x16 are used by the translator only when this says the device reproduces the +// emulator's own conversion. Never fatal: a failure leaves the exact emulation in place. +internal static unsafe class NativeHalfConversionProbe +{ + private const int ReportedExamples = 6; + + // Note carries the reason the probe could not run at all; Detail the first few disagreements. + public readonly record struct Result(bool Exact, int Mismatches, int Tested, string? Note, string? Detail = null); + + // SHARPEMU_NATIVE_HALF=0 forces the emulation, =1 forces native without probing. + public static bool? ReadOverride() => + Environment.GetEnvironmentVariable("SHARPEMU_NATIVE_HALF") switch + { + "0" => false, + "1" => true, + _ => null, + }; + + public static Result Run(GpuDeviceInfo device, SubmissionScheduler scheduler, Func beginCommand) + { + var vectors = HalfConversionReference.BuildTestVectors(); + try + { + var results = Measure(device, scheduler, beginCommand, vectors); + var mismatches = 0; + var examples = new List(ReportedExamples); + for (var index = 0; index < vectors.Length; index++) + { + var wide = HalfConversionReference.HalfToFloatBits(vectors[index] & 0xFFFFu); + if (!HalfConversionReference.Matches(wide, results[2 * index], asHalf: false)) + { + mismatches++; + if (examples.Count < ReportedExamples) + { + examples.Add($"h2f(0x{vectors[index] & 0xFFFF:X4})=0x{results[2 * index]:X8} want 0x{wide:X8}"); + } + } + + var narrow = HalfConversionReference.FloatToHalfBits(vectors[index]); + if (!HalfConversionReference.Matches(narrow, results[2 * index + 1], asHalf: true)) + { + mismatches++; + if (examples.Count < ReportedExamples) + { + examples.Add($"f2h(0x{vectors[index]:X8})=0x{results[2 * index + 1]:X4} want 0x{narrow:X4}"); + } + } + } + + return new Result( + mismatches == 0, + mismatches, + 2 * vectors.Length, + null, + examples.Count == 0 ? null : string.Join("; ", examples)); + } + catch (Exception exception) + { + return new Result(false, -1, 2 * vectors.Length, exception.Message); + } + } + + private static uint[] Measure( + GpuDeviceInfo device, + SubmissionScheduler scheduler, + Func beginCommand, + uint[] vectors) + { + var vk = device.Vk; + var inputBytes = (ulong)vectors.Length * sizeof(uint); + var outputBytes = 2 * inputBytes; + + using var input = new GpuBuffer(device, scheduler, GpuBufferUsage.Upload, 0, GpuBuffer.AllFlags, inputBytes); + using var output = new GpuBuffer(device, scheduler, GpuBufferUsage.Download, 0, GpuBuffer.AllFlags, outputBytes); + MemoryMarshal.AsBytes(vectors).CopyTo(input.Mapped); + input.Flush(0, inputBytes); + output.Mapped.Clear(); + output.Flush(0, outputBytes); + + var bindings = stackalloc DescriptorSetLayoutBinding[2]; + for (uint index = 0; index < 2; index++) + { + bindings[index] = new DescriptorSetLayoutBinding + { + Binding = index, + DescriptorType = DescriptorType.StorageBuffer, + DescriptorCount = 1, + StageFlags = ShaderStageFlags.ComputeBit, + }; + } + + var layoutInfo = new DescriptorSetLayoutCreateInfo + { + SType = StructureType.DescriptorSetLayoutCreateInfo, + BindingCount = 2, + PBindings = bindings, + }; + Require(vk.CreateDescriptorSetLayout(device.Device, &layoutInfo, null, out var setLayout), "descriptor layout"); + + DescriptorPool pool = default; + PipelineLayout pipelineLayout = default; + Pipeline pipeline = default; + try + { + var poolSize = new DescriptorPoolSize { Type = DescriptorType.StorageBuffer, DescriptorCount = 2 }; + var poolInfo = new DescriptorPoolCreateInfo + { + SType = StructureType.DescriptorPoolCreateInfo, + MaxSets = 1, + PoolSizeCount = 1, + PPoolSizes = &poolSize, + }; + Require(vk.CreateDescriptorPool(device.Device, &poolInfo, null, out pool), "descriptor pool"); + + var boundLayout = setLayout; + var allocateInfo = new DescriptorSetAllocateInfo + { + SType = StructureType.DescriptorSetAllocateInfo, + DescriptorPool = pool, + DescriptorSetCount = 1, + PSetLayouts = &boundLayout, + }; + Require(vk.AllocateDescriptorSets(device.Device, &allocateInfo, out DescriptorSet set), "descriptor set"); + + var infos = stackalloc DescriptorBufferInfo[2]; + infos[0] = new DescriptorBufferInfo(input.Handle, 0, inputBytes); + infos[1] = new DescriptorBufferInfo(output.Handle, 0, outputBytes); + var writes = stackalloc WriteDescriptorSet[2]; + for (uint binding = 0; binding < 2; binding++) + { + writes[binding] = new WriteDescriptorSet + { + SType = StructureType.WriteDescriptorSet, + DstSet = set, + DstBinding = binding, + DescriptorCount = 1, + DescriptorType = DescriptorType.StorageBuffer, + PBufferInfo = infos + binding, + }; + } + + vk.UpdateDescriptorSets(device.Device, 2, writes, 0, null); + + var pipelineLayoutInfo = new PipelineLayoutCreateInfo + { + SType = StructureType.PipelineLayoutCreateInfo, + SetLayoutCount = 1, + PSetLayouts = &boundLayout, + }; + Require(vk.CreatePipelineLayout(device.Device, &pipelineLayoutInfo, null, out pipelineLayout), "pipeline layout"); + + var spirv = SpirvFixedShaders.CreateHalfConversionProbe(); + ShaderModule module; + fixed (byte* code = spirv) + { + var moduleInfo = new ShaderModuleCreateInfo + { + SType = StructureType.ShaderModuleCreateInfo, + CodeSize = (nuint)spirv.Length, + PCode = (uint*)code, + }; + Require(vk.CreateShaderModule(device.Device, &moduleInfo, null, out module), "shader module"); + } + + ReadOnlySpan entryPoint = "main\0"u8; + Silk.NET.Vulkan.Result created; + fixed (byte* entry = entryPoint) + { + var pipelineInfo = new ComputePipelineCreateInfo + { + SType = StructureType.ComputePipelineCreateInfo, + Layout = pipelineLayout, + Stage = new PipelineShaderStageCreateInfo + { + SType = StructureType.PipelineShaderStageCreateInfo, + Stage = ShaderStageFlags.ComputeBit, + Module = module, + PName = entry, + }, + }; + created = vk.CreateComputePipelines(device.Device, default, 1, &pipelineInfo, null, out pipeline); + } + + vk.DestroyShaderModule(device.Device, module, null); + Require(created, "compute pipeline"); + + var command = beginCommand(); + vk.CmdBindPipeline(command, PipelineBindPoint.Compute, pipeline); + var boundSet = set; + vk.CmdBindDescriptorSets(command, PipelineBindPoint.Compute, pipelineLayout, 0, 1, &boundSet, 0, null); + vk.CmdDispatch(command, (uint)vectors.Length / 64, 1, 1); + var barrier = new BufferMemoryBarrier2 + { + SType = StructureType.BufferMemoryBarrier2, + SrcAccessMask = AccessFlags2.ShaderWriteBit, + DstAccessMask = AccessFlags2.HostReadBit, + SrcQueueFamilyIndex = Vk.QueueFamilyIgnored, + DstQueueFamilyIndex = Vk.QueueFamilyIgnored, + Buffer = output.Handle, + Offset = 0, + Size = outputBytes, + }; + VulkanSynchronization.PipelineBarrier(vk, command, + PipelineStageFlags.ComputeShaderBit, PipelineStageFlags.HostBit, DependencyFlags.None, + 0, null, 1, &barrier, 0, null); + scheduler.FlushAndWait(); + + output.Invalidate(0, outputBytes); + return MemoryMarshal.Cast(output.Mapped).ToArray(); + } + finally + { + if (pipeline.Handle != 0) + { + vk.DestroyPipeline(device.Device, pipeline, null); + } + + if (pipelineLayout.Handle != 0) + { + vk.DestroyPipelineLayout(device.Device, pipelineLayout, null); + } + + if (pool.Handle != 0) + { + vk.DestroyDescriptorPool(device.Device, pool, null); + } + + vk.DestroyDescriptorSetLayout(device.Device, setLayout, null); + } + } + + private static void Require(Silk.NET.Vulkan.Result result, string operation) + { + if (result != Silk.NET.Vulkan.Result.Success) + { + throw new InvalidOperationException($"The f16 conversion probe could not create its {operation}: {result}"); + } + } +} diff --git a/src/SharpEmu.Libs/Gpu/Vulkan/VulkanAsyncReadback.cs b/src/SharpEmu.Libs/Gpu/Vulkan/VulkanAsyncReadback.cs index 42d8420..353c0b4 100644 --- a/src/SharpEmu.Libs/Gpu/Vulkan/VulkanAsyncReadback.cs +++ b/src/SharpEmu.Libs/Gpu/Vulkan/VulkanAsyncReadback.cs @@ -10,11 +10,27 @@ namespace SharpEmu.Libs.Gpu.Vulkan; public readonly record struct ReadbackPiece(GpuBuffer Source, ulong SourceOffset, ulong Size); +// A readback that waits only for the main-queue tick that last wrote its sources. +internal interface IBufferReadback : IDisposable +{ + void Read(ReadOnlySpan pieces, ulong waitTick, VulkanAsyncReadback.ReadbackConsumer consume); + + bool TryBegin(ReadOnlySpan pieces, ulong waitTick, out VulkanAsyncReadback.Ticket? ticket) + { + ticket = null; + return false; + } + + void Wait(VulkanAsyncReadback.Ticket ticket) => throw new NotSupportedException("This readback backend does not support asynchronous tickets."); + bool IsComplete(VulkanAsyncReadback.Ticket ticket) => throw new NotSupportedException("This readback backend does not support asynchronous tickets."); + void Complete(VulkanAsyncReadback.Ticket ticket, VulkanAsyncReadback.ReadbackConsumer? consume) => throw new NotSupportedException("This readback backend does not support asynchronous tickets."); +} + // Copies GPU-written buffer ranges back to the host on a second queue. The copy waits // on the main queue's timeline for the tick that last wrote the sources only, so a // guest read of one GPU-produced value no longer waits behind every later draw that // is still queued on the main queue (a readback there has to go to the queue's tail). -internal sealed unsafe class VulkanAsyncReadback : IDisposable +internal sealed unsafe class VulkanAsyncReadback : IBufferReadback { private const ulong Alignment = 16; private const int SlotCount = 8; diff --git a/src/SharpEmu.Libs/Gpu/Vulkan/VulkanGuestGpuBackend.cs b/src/SharpEmu.Libs/Gpu/Vulkan/VulkanGuestGpuBackend.cs index 5ca67cc..2539474 100644 --- a/src/SharpEmu.Libs/Gpu/Vulkan/VulkanGuestGpuBackend.cs +++ b/src/SharpEmu.Libs/Gpu/Vulkan/VulkanGuestGpuBackend.cs @@ -38,6 +38,11 @@ internal sealed class VulkanGuestGpuBackend : IGuestGpuBackend public IGuestCompiledShader GetDepthOnlyFragmentShader() => DepthOnlyFragmentShader; + public (IGuestCompiledShader Vertex, IGuestCompiledShader Control) CompileTessellationBridge( + SharpEmu.ShaderCompiler.Resources.BindingLayout layout, Gen5TessellationDomain domain) => + (new VulkanCompiledGuestShader(SpirvFixedShaders.CreateFullscreenVertex(0)), + new VulkanCompiledGuestShader(Gen5TessellationBridge.CompileControl(layout, domain).Spirv)); + public void EnsureStarted(uint width, uint height) => VulkanVideoPresenter.EnsureStarted(width, height); diff --git a/src/SharpEmu.Libs/Gpu/Vulkan/VulkanTickDevice.cs b/src/SharpEmu.Libs/Gpu/Vulkan/VulkanTickDevice.cs index db3e26f..0202b60 100644 --- a/src/SharpEmu.Libs/Gpu/Vulkan/VulkanTickDevice.cs +++ b/src/SharpEmu.Libs/Gpu/Vulkan/VulkanTickDevice.cs @@ -360,11 +360,65 @@ internal sealed unsafe class VulkanTickDevice : IGpuTickDevice if (result == Result.Success) CommandProfile?.MarkSubmitted(buffer); - failure = result.ToString(); + failure = result == Result.ErrorDeviceLost ? $"{result} {DescribeDeviceFault()}" : result.ToString(); return result == Result.Success; } } + // VK_EXT_device_fault: the driver's record of the faulting addresses, when it keeps one. + private string DescribeDeviceFault() + { + var name = System.Text.Encoding.ASCII.GetBytes("vkGetDeviceFaultInfoEXT\0"); + nint function; + fixed (byte* namePointer = name) + { + function = (nint)_vk.GetDeviceProcAddr(_device, namePointer); + } + + if (function == 0) + { + return "fault=unavailable"; + } + + var getFaultInfo = (delegate* unmanaged)function; + var counts = new DeviceFaultCountsEXT { SType = StructureType.DeviceFaultCountsExt }; + if (getFaultInfo(_device, &counts, null) is not (Result.Success or Result.Incomplete)) + { + return "fault=unreadable"; + } + + var addresses = new DeviceFaultAddressInfoEXT[Math.Max(1u, counts.AddressInfoCount)]; + var vendors = new DeviceFaultVendorInfoEXT[Math.Max(1u, counts.VendorInfoCount)]; + counts.VendorBinarySize = 0; + fixed (DeviceFaultAddressInfoEXT* addressPointer = addresses) + fixed (DeviceFaultVendorInfoEXT* vendorPointer = vendors) + { + var info = new DeviceFaultInfoEXT + { + SType = StructureType.DeviceFaultInfoExt, + PAddressInfos = addressPointer, + PVendorInfos = vendorPointer, + }; + getFaultInfo(_device, &counts, &info); + var text = new System.Text.StringBuilder($"fault=\"{System.Runtime.InteropServices.Marshal.PtrToStringUTF8((nint)info.Description)}\""); + for (var index = 0; index < counts.AddressInfoCount; index++) + { + text.Append($" address[{index}]={addresses[index].AddressType}:0x{addresses[index].ReportedAddress:X}/0x{addresses[index].AddressPrecision:X}"); + text.Append(SharpEmu.Libs.Gpu.Buffers.GpuBufferAddressBook.Describe(addresses[index].ReportedAddress)); + } + + for (var index = 0; index < counts.VendorInfoCount; index++) + { + fixed (byte* description = vendors[index].Description) + { + text.Append($" vendor[{index}]=0x{vendors[index].VendorFaultCode:X}:0x{vendors[index].VendorFaultData:X} {System.Runtime.InteropServices.Marshal.PtrToStringUTF8((nint)description)}"); + } + } + + return text.ToString(); + } + } + public void Dispose() { CommandProfile?.Dispose(); diff --git a/src/SharpEmu.Libs/Gpu/Vulkan/ZeroOutOfBoundsReadProbe.cs b/src/SharpEmu.Libs/Gpu/Vulkan/ZeroOutOfBoundsReadProbe.cs new file mode 100644 index 0000000..9caec72 --- /dev/null +++ b/src/SharpEmu.Libs/Gpu/Vulkan/ZeroOutOfBoundsReadProbe.cs @@ -0,0 +1,242 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Runtime.InteropServices; +using SharpEmu.Libs.Gpu.Buffers; +using SharpEmu.Libs.Gpu.Scheduling; +using SharpEmu.ShaderCompiler.Vulkan; +using Silk.NET.Vulkan; + +namespace SharpEmu.Libs.Gpu.Vulkan; + +// Measures whether a storage-buffer read past the end of its descriptor range reads zero on this +// device. Every guest buffer word the translator emits carries a range test, an address clamp and +// a zero select, which exist only to produce that zero; with robustBufferAccess2 the device owes +// it already and the three operations are pure overhead. The capability bit alone does not settle +// it — MoltenVK runs on Metal, which has no hardware bounds checking — so the probe reads a bound +// range it deliberately undershoots and checks the words past it. Never run on a device that did +// not enable robustBufferAccess2: an out-of-range read there is undefined and can fault. +internal static unsafe class ZeroOutOfBoundsReadProbe +{ + // The descriptor binds this many words of a larger allocation; the words past it must read + // zero, and the words inside it must still read the sentinel the probe wrote. + private const uint BoundWords = 256; + private const uint ProbeWords = 1024; + private const uint Sentinel = 0xA5C3_F00Du; + + public readonly record struct Result(bool Zeroed, int Mismatches, int Tested, string? Note); + + // SHARPEMU_ZERO_OOB_READS=0 keeps the per-access checks, =1 drops them without measuring. + public static bool? ReadOverride() => + Environment.GetEnvironmentVariable("SHARPEMU_ZERO_OOB_READS") switch + { + "0" => false, + "1" => true, + _ => null, + }; + + public static Result Run(GpuDeviceInfo device, SubmissionScheduler scheduler, Func beginCommand) + { + try + { + var results = Measure(device, scheduler, beginCommand); + var mismatches = 0; + for (var index = 0u; index < ProbeWords; index++) + { + var expected = index < BoundWords ? Sentinel + index : 0u; + if (results[index] != expected) + { + mismatches++; + } + } + + return new Result(mismatches == 0, mismatches, (int)ProbeWords, null); + } + catch (Exception exception) + { + return new Result(false, -1, (int)ProbeWords, exception.Message); + } + } + + private static uint[] Measure(GpuDeviceInfo device, SubmissionScheduler scheduler, Func beginCommand) + { + var vk = device.Vk; + var allocationBytes = (ulong)ProbeWords * sizeof(uint); + var boundBytes = (ulong)BoundWords * sizeof(uint); + var outputBytes = allocationBytes; + + // The allocation is larger than the bound range and carries a sentinel everywhere, so a + // device that simply ignores the range reads the sentinel back instead of zero. + using var input = new GpuBuffer(device, scheduler, GpuBufferUsage.Upload, 0, GpuBuffer.AllFlags, allocationBytes); + using var output = new GpuBuffer(device, scheduler, GpuBufferUsage.Download, 0, GpuBuffer.AllFlags, outputBytes); + var words = new uint[ProbeWords]; + for (var index = 0u; index < ProbeWords; index++) + { + words[index] = Sentinel + index; + } + + MemoryMarshal.AsBytes(words).CopyTo(input.Mapped); + input.Flush(0, allocationBytes); + output.Mapped.Clear(); + output.Flush(0, outputBytes); + + var bindings = stackalloc DescriptorSetLayoutBinding[2]; + for (uint index = 0; index < 2; index++) + { + bindings[index] = new DescriptorSetLayoutBinding + { + Binding = index, + DescriptorType = DescriptorType.StorageBuffer, + DescriptorCount = 1, + StageFlags = ShaderStageFlags.ComputeBit, + }; + } + + var layoutInfo = new DescriptorSetLayoutCreateInfo + { + SType = StructureType.DescriptorSetLayoutCreateInfo, + BindingCount = 2, + PBindings = bindings, + }; + Require(vk.CreateDescriptorSetLayout(device.Device, &layoutInfo, null, out var setLayout), "descriptor layout"); + + DescriptorPool pool = default; + PipelineLayout pipelineLayout = default; + Pipeline pipeline = default; + try + { + var poolSize = new DescriptorPoolSize { Type = DescriptorType.StorageBuffer, DescriptorCount = 2 }; + var poolInfo = new DescriptorPoolCreateInfo + { + SType = StructureType.DescriptorPoolCreateInfo, + MaxSets = 1, + PoolSizeCount = 1, + PPoolSizes = &poolSize, + }; + Require(vk.CreateDescriptorPool(device.Device, &poolInfo, null, out pool), "descriptor pool"); + + var boundLayout = setLayout; + var allocateInfo = new DescriptorSetAllocateInfo + { + SType = StructureType.DescriptorSetAllocateInfo, + DescriptorPool = pool, + DescriptorSetCount = 1, + PSetLayouts = &boundLayout, + }; + Require(vk.AllocateDescriptorSets(device.Device, &allocateInfo, out DescriptorSet set), "descriptor set"); + + var infos = stackalloc DescriptorBufferInfo[2]; + infos[0] = new DescriptorBufferInfo(input.Handle, 0, boundBytes); + infos[1] = new DescriptorBufferInfo(output.Handle, 0, outputBytes); + var writes = stackalloc WriteDescriptorSet[2]; + for (uint binding = 0; binding < 2; binding++) + { + writes[binding] = new WriteDescriptorSet + { + SType = StructureType.WriteDescriptorSet, + DstSet = set, + DstBinding = binding, + DescriptorCount = 1, + DescriptorType = DescriptorType.StorageBuffer, + PBufferInfo = infos + binding, + }; + } + + vk.UpdateDescriptorSets(device.Device, 2, writes, 0, null); + + var pipelineLayoutInfo = new PipelineLayoutCreateInfo + { + SType = StructureType.PipelineLayoutCreateInfo, + SetLayoutCount = 1, + PSetLayouts = &boundLayout, + }; + Require(vk.CreatePipelineLayout(device.Device, &pipelineLayoutInfo, null, out pipelineLayout), "pipeline layout"); + + var spirv = SpirvFixedShaders.CreateUncheckedBufferCopyProbe(); + ShaderModule module; + fixed (byte* code = spirv) + { + var moduleInfo = new ShaderModuleCreateInfo + { + SType = StructureType.ShaderModuleCreateInfo, + CodeSize = (nuint)spirv.Length, + PCode = (uint*)code, + }; + Require(vk.CreateShaderModule(device.Device, &moduleInfo, null, out module), "shader module"); + } + + ReadOnlySpan entryPoint = "main\0"u8; + Silk.NET.Vulkan.Result created; + fixed (byte* entry = entryPoint) + { + var pipelineInfo = new ComputePipelineCreateInfo + { + SType = StructureType.ComputePipelineCreateInfo, + Layout = pipelineLayout, + Stage = new PipelineShaderStageCreateInfo + { + SType = StructureType.PipelineShaderStageCreateInfo, + Stage = ShaderStageFlags.ComputeBit, + Module = module, + PName = entry, + }, + }; + created = vk.CreateComputePipelines(device.Device, default, 1, &pipelineInfo, null, out pipeline); + } + + vk.DestroyShaderModule(device.Device, module, null); + Require(created, "compute pipeline"); + + var command = beginCommand(); + vk.CmdBindPipeline(command, PipelineBindPoint.Compute, pipeline); + var boundSet = set; + vk.CmdBindDescriptorSets(command, PipelineBindPoint.Compute, pipelineLayout, 0, 1, &boundSet, 0, null); + vk.CmdDispatch(command, ProbeWords / 64, 1, 1); + var barrier = new BufferMemoryBarrier2 + { + SType = StructureType.BufferMemoryBarrier2, + SrcAccessMask = AccessFlags2.ShaderWriteBit, + DstAccessMask = AccessFlags2.HostReadBit, + SrcQueueFamilyIndex = Vk.QueueFamilyIgnored, + DstQueueFamilyIndex = Vk.QueueFamilyIgnored, + Buffer = output.Handle, + Offset = 0, + Size = outputBytes, + }; + VulkanSynchronization.PipelineBarrier(vk, command, + PipelineStageFlags.ComputeShaderBit, PipelineStageFlags.HostBit, DependencyFlags.None, + 0, null, 1, &barrier, 0, null); + scheduler.FlushAndWait(); + + output.Invalidate(0, outputBytes); + return MemoryMarshal.Cast(output.Mapped).ToArray(); + } + finally + { + if (pipeline.Handle != 0) + { + vk.DestroyPipeline(device.Device, pipeline, null); + } + + if (pipelineLayout.Handle != 0) + { + vk.DestroyPipelineLayout(device.Device, pipelineLayout, null); + } + + if (pool.Handle != 0) + { + vk.DestroyDescriptorPool(device.Device, pool, null); + } + + vk.DestroyDescriptorSetLayout(device.Device, setLayout, null); + } + } + + private static void Require(Silk.NET.Vulkan.Result result, string operation) + { + if (result != Silk.NET.Vulkan.Result.Success) + { + throw new InvalidOperationException($"The out-of-range read probe could not create its {operation}: {result}"); + } + } +} diff --git a/src/SharpEmu.Libs/Kernel/KernelEventFlagCompatExports.cs b/src/SharpEmu.Libs/Kernel/KernelEventFlagCompatExports.cs index cb1edc7..60ddb9c 100644 --- a/src/SharpEmu.Libs/Kernel/KernelEventFlagCompatExports.cs +++ b/src/SharpEmu.Libs/Kernel/KernelEventFlagCompatExports.cs @@ -276,13 +276,17 @@ public static class KernelEventFlagCompatExports return (int)blockedWaitResult; } - // Deadline expiry: report timeout with the current bits. - if (timeoutAddress != 0) + // Deadline expiry releases this waiter just as a successful wake does. + lock (state.Gate) { - _ = TryWriteUInt32(ctx, timeoutAddress, 0); - } + state.WaitingThreads = Math.Max(0, state.WaitingThreads - 1); + if (timeoutAddress != 0) + { + _ = TryWriteUInt32(ctx, timeoutAddress, 0); + } - _ = TryWriteResultPattern(ctx, resultAddress, state.Bits); + _ = TryWriteResultPattern(ctx, resultAddress, state.Bits); + } return (int)OrbisGen2Result.ORBIS_GEN2_ERROR_TIMED_OUT; }, () => diff --git a/src/SharpEmu.Libs/Kernel/KernelFileExtendedExports.cs b/src/SharpEmu.Libs/Kernel/KernelFileExtendedExports.cs index b4887f6..456df86 100644 --- a/src/SharpEmu.Libs/Kernel/KernelFileExtendedExports.cs +++ b/src/SharpEmu.Libs/Kernel/KernelFileExtendedExports.cs @@ -619,6 +619,10 @@ public static partial class KernelMemoryCompatExports Target = Generation.Gen4 | Generation.Gen5, LibraryName = "libKernel")] public static int KernelAioPollRequests(CpuContext ctx) => KernelAioComplete(ctx); + [SysAbiExport(Nid = "2pOuoWoCxdk", ExportName = "sceKernelAioPollRequest", + Target = Generation.Gen4 | Generation.Gen5, LibraryName = "libKernel")] + public static int KernelAioPollRequest(CpuContext ctx) => KernelAioGetRequestState(ctx); + [SysAbiExport(Nid = "lgK+oIWkJyA", ExportName = "sceKernelAioWaitRequests", Target = Generation.Gen4 | Generation.Gen5, LibraryName = "libKernel")] public static int KernelAioWaitRequests(CpuContext ctx) @@ -633,7 +637,9 @@ public static partial class KernelMemoryCompatExports [SysAbiExport(Nid = "KOF-oJbQVvc", ExportName = "sceKernelAioWaitRequest", Target = Generation.Gen4 | Generation.Gen5, LibraryName = "libKernel")] - public static int KernelAioWaitRequest(CpuContext ctx) + public static int KernelAioWaitRequest(CpuContext ctx) => KernelAioGetRequestState(ctx); + + private static int KernelAioGetRequestState(CpuContext ctx) { var submitId = unchecked((uint)ctx[CpuRegister.Rdi]); if (!_aioResults.TryGetValue(submitId, out var completedState)) diff --git a/src/SharpEmu.Libs/Kernel/KernelMemoryCompatExports.BackedMappings.cs b/src/SharpEmu.Libs/Kernel/KernelMemoryCompatExports.BackedMappings.cs index 80fae25..2726994 100644 --- a/src/SharpEmu.Libs/Kernel/KernelMemoryCompatExports.BackedMappings.cs +++ b/src/SharpEmu.Libs/Kernel/KernelMemoryCompatExports.BackedMappings.cs @@ -408,10 +408,23 @@ public static partial class KernelMemoryCompatExports { _insideMappingTransaction = false; } - }); + }, () => onlyAddsOutsideGpuMemory?.Invoke() != true); return result; } + // Whether a map request may replace memory the GPU can be using. Without MAP_FIXED the address + // is chosen among free or reserved ranges; with it, only a live (non-reserved) mapping in the + // range can be in use. An unreadable request pointer keeps the safe answer. + private static Func MapNeedsGpuDrain(CpuContext ctx, ulong addressPointer, ulong length, ulong flags) => () => + { + if ((flags & OrbisKernelMapFixed) == 0) + return false; + if (!ctx.TryReadUInt64(addressPointer, out var requested) || length == 0 || requested > ulong.MaxValue - length) + return true; + lock (_memoryGate) + return GetMappingSlices(requested, length).Any(region => !region.IsReserved); + }; + private static int ReserveBackingRangeCore(CpuContext ctx, ulong pointer, ulong length, ulong flags, ulong alignment) { if (pointer == 0 || !IsValidMapRange(length, alignment)) diff --git a/src/SharpEmu.Libs/Kernel/KernelMemoryCompatExports.cs b/src/SharpEmu.Libs/Kernel/KernelMemoryCompatExports.cs index f757baf..c815660 100644 --- a/src/SharpEmu.Libs/Kernel/KernelMemoryCompatExports.cs +++ b/src/SharpEmu.Libs/Kernel/KernelMemoryCompatExports.cs @@ -98,7 +98,7 @@ public static partial class KernelMemoryCompatExports private static readonly object _fdGate = new(); private static readonly Dictionary _openFiles = new(); - private static readonly HashSet _randomDeviceDescriptors = new(); + private static readonly HashSet _entropyDevices = new(); private static readonly Dictionary _binkGuestCompletionShims = new(); private static readonly Dictionary _observedBinkGuestFiles = new(); @@ -1455,16 +1455,19 @@ public static partial class KernelMemoryCompatExports return (int)OrbisGen2Result.ORBIS_GEN2_ERROR_MEMORY_FAULT; } - if (IsRandomDevicePath(guestPath) && (flags & O_DIRECTORY) == 0) + if (guestPath is "/dev/urandom" or "/dev/random") { - var randomFd = (int)Interlocked.Increment(ref _nextFileDescriptor); + // These character devices provide entropy, not mounted host files. + // Only read-only opens are currently supported. + if (ResolveOpenAccess(flags) != FileAccess.Read || + (flags & (O_CREAT | O_TRUNC | O_APPEND | O_DIRECTORY)) != 0) + return (int)OrbisGen2Result.ORBIS_GEN2_ERROR_INVALID_ARGUMENT; lock (_fdGate) { - _randomDeviceDescriptors.Add(randomFd); + var entropyFd = (int)Interlocked.Increment(ref _nextFileDescriptor); + _entropyDevices.Add(entropyFd); + ctx[CpuRegister.Rax] = unchecked((ulong)entropyFd); } - - LogOpenTrace($"_open random-device path='{guestPath}' flags=0x{flags:X8} fd={randomFd}"); - ctx[CpuRegister.Rax] = unchecked((ulong)randomFd); return (int)OrbisGen2Result.ORBIS_GEN2_OK; } @@ -2273,12 +2276,12 @@ public static partial class KernelMemoryCompatExports string? observedBinkPath = null; lock (_fdGate) { - if (_randomDeviceDescriptors.Remove(fd)) + if (_entropyDevices.Remove(fd)) { ctx[CpuRegister.Rax] = 0; return (int)OrbisGen2Result.ORBIS_GEN2_OK; } - else if (_openFiles.Remove(fd, out stream)) + if (_openFiles.Remove(fd, out stream)) { _binkGuestCompletionShims.Remove(fd); if (_observedBinkGuestFiles.Remove(fd, out observedBinkPath)) @@ -2331,23 +2334,17 @@ public static partial class KernelMemoryCompatExports return (int)OrbisGen2Result.ORBIS_GEN2_OK; } - bool isRandomDevice; lock (_fdGate) { - isRandomDevice = _randomDeviceDescriptors.Contains(fd); - } - - if (isRandomDevice) - { - var randomBytes = GC.AllocateUninitializedArray(requested); - RandomNumberGenerator.Fill(randomBytes); - if (!ctx.Memory.TryWrite(bufferAddress, randomBytes)) + if (_entropyDevices.Contains(fd)) { - return (int)OrbisGen2Result.ORBIS_GEN2_ERROR_MEMORY_FAULT; + var bytes = GC.AllocateUninitializedArray(requested); + System.Security.Cryptography.RandomNumberGenerator.Fill(bytes); + if (!TryWriteCompat(ctx, bufferAddress, bytes)) + return (int)OrbisGen2Result.ORBIS_GEN2_ERROR_MEMORY_FAULT; + ctx[CpuRegister.Rax] = unchecked((ulong)requested); + return (int)OrbisGen2Result.ORBIS_GEN2_OK; } - - ctx[CpuRegister.Rax] = unchecked((ulong)requested); - return (int)OrbisGen2Result.ORBIS_GEN2_OK; } if (KernelSocketCompatExports.TryReadSocketFd( @@ -2401,41 +2398,53 @@ public static partial class KernelMemoryCompatExports positionBefore = -1; } - var buffer = GC.AllocateUninitializedArray(requested); - var read = stream.Read(buffer, 0, requested); - if (read > 0 && useBinkCompletionShim) - { - // The patched NumFrames field is what tells the guest "this - // movie is fully consumed" - hold that specific read until the - // host has actually finished showing it, so guest-side game - // logic can't race ahead of what's still on screen. - if (completionShim.Patch(positionBefore, buffer.AsSpan(0, read))) - { - HostMovieBridge.WaitForHostPlaybackToFinish(stream.Name); - } - } - if (read > 0 && !ctx.Memory.TryWrite(bufferAddress, buffer.AsSpan(0, read))) - { - return (int)OrbisGen2Result.ORBIS_GEN2_ERROR_MEMORY_FAULT; - } - - long positionAfter; + // Guest reads are frequent and mostly small: a pooled buffer spares each one an allocation. + var pooled = requested <= MaxPooledReadBytes; + var buffer = pooled ? ArrayPool.Shared.Rent(requested) : GC.AllocateUninitializedArray(requested); try { - positionAfter = stream.Position; + var read = stream.Read(buffer, 0, requested); + if (read > 0 && useBinkCompletionShim) + { + // The patched NumFrames field is what tells the guest "this + // movie is fully consumed" - hold that specific read until the + // host has actually finished showing it, so guest-side game + // logic can't race ahead of what's still on screen. + if (completionShim.Patch(positionBefore, buffer.AsSpan(0, read))) + { + HostMovieBridge.WaitForHostPlaybackToFinish(stream.Name); + } + } + if (read > 0 && !ctx.Memory.TryWrite(bufferAddress, buffer.AsSpan(0, read))) + { + return (int)OrbisGen2Result.ORBIS_GEN2_ERROR_MEMORY_FAULT; + } + + long positionAfter; + try + { + positionAfter = stream.Position; + } + catch (IOException) + { + positionAfter = -1; + } + + LogIoTrace( + "read", + stream.Name, + $"fd={fd} req={requested} read={read} pos={positionBefore}->{positionAfter} preview='{PreviewIoBytes(buffer, read, 64)}' hex={PreviewIoHex(buffer, read, 32)} guest_tail={PreviewGuestHex(ctx, bufferAddress + (ulong)Math.Max(read, 0), 32)}"); + + ctx[CpuRegister.Rax] = unchecked((ulong)read); + return (int)OrbisGen2Result.ORBIS_GEN2_OK; } - catch (IOException) + finally { - positionAfter = -1; + if (pooled) + { + ArrayPool.Shared.Return(buffer); + } } - - LogIoTrace( - "read", - stream.Name, - $"fd={fd} req={requested} read={read} pos={positionBefore}->{positionAfter} preview='{PreviewIoBytes(buffer, read, 64)}' hex={PreviewIoHex(buffer, read, 32)} guest_tail={PreviewGuestHex(ctx, bufferAddress + (ulong)Math.Max(read, 0), 32)}"); - - ctx[CpuRegister.Rax] = unchecked((ulong)read); - return (int)OrbisGen2Result.ORBIS_GEN2_OK; } [SysAbiExport( @@ -3329,7 +3338,8 @@ public static partial class KernelMemoryCompatExports Target = Generation.Gen4 | Generation.Gen5, LibraryName = "libKernel")] public static int KernelMapNamedFlexibleMemory(CpuContext ctx) - => RunMappingTransaction(() => MapFlexibleMemoryCore(ctx)); + => RunMappingTransaction(() => MapFlexibleMemoryCore(ctx), + () => IsNewMappingOutsideGpuMemory(ctx, ctx[CpuRegister.Rdi], ctx[CpuRegister.Rsi], ctx[CpuRegister.Rcx])); private static int MapFlexibleMemoryCore(CpuContext ctx) { @@ -6368,6 +6378,25 @@ public static partial class KernelMemoryCompatExports : HostPageNoAccess; } + // The guest mapping containing the address, for the GPU caches: a device-address fault + // inside it registers part of that mapping rather than a single page. + internal static bool TryGetMappedRange(ulong address, out ulong start, out ulong length) + { + lock (_memoryGate) + { + if (TryFindVirtualQueryRegionLocked(address, findNext: false, out var region) && !region.IsReserved) + { + start = region.Address; + length = region.Length; + return true; + } + } + + start = 0; + length = 0; + return false; + } + private static bool TryFindVirtualQueryRegionLocked(ulong queryAddress, bool findNext, out MappedRegion region) { region = default; @@ -6952,14 +6981,15 @@ public static partial class KernelMemoryCompatExports string? hostPath = null; bool isDirectory = false; - bool isRandomDevice = false; lock (_fdGate) { - if (_randomDeviceDescriptors.Contains(fd)) + if (_entropyDevices.Contains(fd)) { - isRandomDevice = true; + return TryWriteKernelStat(ctx, statAddress, isDirectory: false, size: 0, + DateTime.UnixEpoch, DateTime.UnixEpoch, DateTime.UnixEpoch, + "entropy-device", modeOverride: 0x2124); } - else if (_openDirectories.TryGetValue(fd, out var directory)) + if (_openDirectories.TryGetValue(fd, out var directory)) { hostPath = directory.Path; isDirectory = true; @@ -6970,14 +7000,6 @@ public static partial class KernelMemoryCompatExports } } - if (isRandomDevice) - { - var now = DateTime.UtcNow; - LogIoTrace("fstat", "/dev/urandom", $"fd={fd} size=0 device=random"); - return TryWriteKernelStat(ctx, statAddress, isDirectory: false, size: 0, - now, now, now, "/dev/urandom"); - } - if (!string.IsNullOrWhiteSpace(hostPath)) { long size = 0; @@ -6999,10 +7021,6 @@ public static partial class KernelMemoryCompatExports return !string.IsNullOrWhiteSpace(hostPath) && TryWriteHostPathStat(ctx, statAddress, hostPath!, isDirectory); } - private static bool IsRandomDevicePath(string path) => - string.Equals(path, "/dev/random", StringComparison.Ordinal) || - string.Equals(path, "/dev/urandom", StringComparison.Ordinal); - private static bool TryWriteHostPathStat(CpuContext ctx, ulong statAddress, string hostPath) { var isDirectory = Directory.Exists(hostPath); @@ -7118,7 +7136,8 @@ public static partial class KernelMemoryCompatExports DateTime lastAccessUtc, DateTime lastWriteUtc, DateTime creationUtc, - string inodeSeed) + string inodeSeed, + ushort? modeOverride = null) { Span payload = stackalloc byte[KernelStatSize]; payload.Clear(); @@ -7126,7 +7145,7 @@ public static partial class KernelMemoryCompatExports var seedBytes = Encoding.UTF8.GetBytes(inodeSeed); BinaryPrimitives.WriteUInt32LittleEndian(payload[KernelStatStDevOffset..], 0); BinaryPrimitives.WriteUInt32LittleEndian(payload[KernelStatStInoOffset..], ComputeDirectoryEntryHash(seedBytes)); - BinaryPrimitives.WriteUInt16LittleEndian(payload[KernelStatStModeOffset..], isDirectory ? KernelStatModeDirectory : KernelStatModeRegular); + BinaryPrimitives.WriteUInt16LittleEndian(payload[KernelStatStModeOffset..], modeOverride ?? (isDirectory ? KernelStatModeDirectory : KernelStatModeRegular)); BinaryPrimitives.WriteUInt16LittleEndian(payload[KernelStatStNlinkOffset..], 1); BinaryPrimitives.WriteUInt32LittleEndian(payload[KernelStatStUidOffset..], 0); BinaryPrimitives.WriteUInt32LittleEndian(payload[KernelStatStGidOffset..], 0); @@ -7274,9 +7293,41 @@ public static partial class KernelMemoryCompatExports } // Read once: file-heavy titles call LogIoTrace for every resolved path. + private const int MaxPooledReadBytes = 1 << 20; + private static readonly bool _logIo = string.Equals(Environment.GetEnvironmentVariable("SHARPEMU_LOG_IO"), "1", StringComparison.Ordinal); + // Builds an I/O trace detail only while SHARPEMU_LOG_IO is on: its holes (byte previews of + // every read) are otherwise never evaluated. + [System.Runtime.CompilerServices.InterpolatedStringHandler] + private ref struct IoTraceDetail + { + private System.Runtime.CompilerServices.DefaultInterpolatedStringHandler _inner; + + public IoTraceDetail(int literalLength, int formattedCount, out bool enabled) + { + enabled = _logIo; + _inner = enabled ? new System.Runtime.CompilerServices.DefaultInterpolatedStringHandler(literalLength, formattedCount) : default; + } + + public void AppendLiteral(string value) => _inner.AppendLiteral(value); + + public void AppendFormatted(T value) => _inner.AppendFormatted(value); + + public void AppendFormatted(T value, string? format) => _inner.AppendFormatted(value, format); + + public string ToStringAndClear() => _inner.ToStringAndClear(); + } + + private static void LogIoTrace(string operation, string path, IoTraceDetail detail) + { + if (_logIo) + { + LogIoTrace(operation, path, detail.ToStringAndClear()); + } + } + private static void LogIoTrace(string operation, string path, string detail) { if (!_logIo) diff --git a/src/SharpEmu.Libs/Kernel/KernelPthreadCompatExports.cs b/src/SharpEmu.Libs/Kernel/KernelPthreadCompatExports.cs index 3cea481..4004ec0 100644 --- a/src/SharpEmu.Libs/Kernel/KernelPthreadCompatExports.cs +++ b/src/SharpEmu.Libs/Kernel/KernelPthreadCompatExports.cs @@ -165,6 +165,8 @@ public static class KernelPthreadCompatExports public LinkedListNode? Node { get; set; } public PthreadMutexWaiter? MutexWaiter { get; set; } public Timer? TimeoutTimer { get; set; } + // Recursion depth of the mutex before the wait released it; restored on wakeup. + public int SavedRecursion { get; init; } = 1; // 0 = waiting, 1 = signaled, 2 = timed out. public int CompletionState { get; set; } } @@ -376,6 +378,33 @@ public static class KernelPthreadCompatExports LibraryName = "libKernel")] public static int PthreadMutexTrylock(CpuContext ctx) => PthreadMutexLockCore(ctx, ctx[CpuRegister.Rdi], tryOnly: true); + [SysAbiExport( + Nid = "IafI2PxcPnQ", + ExportName = "scePthreadMutexTimedlock", + Target = Generation.Gen4 | Generation.Gen5, + LibraryName = "libKernel")] + public static int PthreadMutexTimedlock(CpuContext ctx) + { + // Unlike POSIX pthread_mutex_timedlock's absolute timespec, this export + // receives a 32-bit relative microsecond interval in the second argument. + var mutexAddress = ctx[CpuRegister.Rdi]; + var timeoutUsec = unchecked((uint)ctx[CpuRegister.Rsi]); + var deadline = GuestThreadExecution.ComputeDeadlineTimestamp(TimeSpan.FromTicks((long)timeoutUsec * 10)); + while (true) + { + var result = PthreadMutexLockCore(ctx, mutexAddress, tryOnly: true); + if (result != (int)OrbisGen2Result.ORBIS_GEN2_ERROR_BUSY) + return result; + if (Stopwatch.GetTimestamp() >= deadline) + return (int)OrbisGen2Result.ORBIS_GEN2_ERROR_TIMED_OUT; + + // Recheck ownership after every wake. A timeout never grants the lock + // or leaves a queued waiter behind. Trylock also preserves recursion. + Thread.Sleep(1); + GuestThreadExecution.Scheduler?.DeliverPendingGuestExceptionIfReady(ctx); + } + } + [SysAbiExport( Nid = "tn3VlD0hG60", ExportName = "scePthreadMutexUnlock", @@ -1054,12 +1083,7 @@ public static class KernelPthreadCompatExports } if (canCooperativelyBlock && waiter is not null && - GuestThreadExecution.RequestCurrentThreadBlock( - ctx, - "pthread_mutex_lock", - waiter.WakeKey, - () => CompleteBlockedMutexLock(ctx, mutexAddress, resolvedAddress, state, waiter), - () => TryGrantBlockedMutexLock(ctx, mutexAddress, resolvedAddress, state, waiter))) + RequestBlockedMutexLock(ctx, mutexAddress, resolvedAddress, state, waiter)) { TracePthreadMutex(ctx, "lock-block", mutexAddress, resolvedAddress, state, currentThreadId, (int)OrbisGen2Result.ORBIS_GEN2_OK); return (int)OrbisGen2Result.ORBIS_GEN2_OK; @@ -1070,6 +1094,21 @@ public static class KernelPthreadCompatExports return hostResult; } + // The continuations capture the lock's arguments in a closure; keeping them out of + // PthreadMutexLockCore spares every uncontended lock that allocation. + private static bool RequestBlockedMutexLock( + CpuContext ctx, + ulong mutexAddress, + ulong resolvedAddress, + PthreadMutexState state, + PthreadMutexWaiter waiter) => + GuestThreadExecution.RequestCurrentThreadBlock( + ctx, + "pthread_mutex_lock", + waiter.WakeKey, + () => CompleteBlockedMutexLock(ctx, mutexAddress, resolvedAddress, state, waiter), + () => TryGrantBlockedMutexLock(ctx, mutexAddress, resolvedAddress, state, waiter)); + private static int PthreadMutexUnlockCore(CpuContext ctx, ulong mutexAddress, bool requireOwner) { if (mutexAddress == 0) @@ -1895,6 +1934,7 @@ public static class KernelPthreadCompatExports } var currentThreadId = KernelPthreadState.GetCurrentThreadHandle(); + var savedRecursion = 1; lock (mutexState.SyncRoot) { if (mutexState.OwnerThreadId == 0 && mutexState.RecursionCount == 0) @@ -1911,6 +1951,15 @@ public static class KernelPthreadCompatExports _ = mutexState.TryAcquireOwner(currentThreadId); } + // libthr's mutex_cv_unlock releases a recursive mutex completely and + // mutex_cv_lock restores its depth, so a nested lock may wait. + if (mutexState.OwnerThreadId == currentThreadId && mutexState.RecursionCount > 1 && + mutexState.Type == MutexTypeRecursive) + { + savedRecursion = mutexState.RecursionCount; + mutexState.RecursionCount = 1; + } + if (mutexState.OwnerThreadId != currentThreadId || mutexState.RecursionCount != 1) { return mutexState.OwnerThreadId == currentThreadId @@ -1927,6 +1976,7 @@ public static class KernelPthreadCompatExports MutexState = mutexState, Cooperative = cooperative, PosixErrors = posixErrors, + SavedRecursion = savedRecursion, WakeKey = cooperative ? $"pthread_cond_waiter:{Interlocked.Increment(ref _nextSynchronizationWaiterId)}" : string.Empty, @@ -1942,6 +1992,7 @@ public static class KernelPthreadCompatExports if (unlockResult != (int)OrbisGen2Result.ORBIS_GEN2_OK) { RemoveCondWaiterLocked(state, waiter); + RestoreCondWaiterRecursion(waiter); TracePthreadCond("wait-unlock-fail", condAddress, mutexAddress, state, timed, unlockResult); return unlockResult; } @@ -1988,7 +2039,7 @@ public static class KernelPthreadCompatExports break; } - var waitDuration = TimeSpan.FromMilliseconds(10); + var waitMilliseconds = 10; if (timed) { var remaining = GetRemainingTimeout(deadline); @@ -1998,13 +2049,14 @@ public static class KernelPthreadCompatExports break; } - if (remaining < waitDuration) - { - waitDuration = remaining; - } + // Monitor.Wait truncates to whole milliseconds, so a sub-millisecond remainder + // became a zero timeout and the wait spun until the deadline. UE's game thread + // waits for the render fence in 1 ms slices, which kept a core busy. A timed wait + // may wake late, so round up instead. + waitMilliseconds = (int)Math.Min(waitMilliseconds, Math.Ceiling(remaining.TotalMilliseconds)); } - _ = Monitor.Wait(state.SyncRoot, waitDuration); + _ = Monitor.Wait(state.SyncRoot, waitMilliseconds); if (waiter.CompletionState == 0 && timed && GetRemainingTimeout(deadline) <= TimeSpan.Zero) @@ -2035,6 +2087,7 @@ public static class KernelPthreadCompatExports } _ = WaitForHostMutexLock(ctx, mutexState, waiter.MutexWaiter); + RestoreCondWaiterRecursion(waiter); var waitResult = waiter.CompletionState == 2 ? CondTimedOutResult(waiter) : (int)OrbisGen2Result.ORBIS_GEN2_OK; @@ -2454,6 +2507,11 @@ public static class KernelPthreadCompatExports ? CondTimedOutResult(waiter) : (int)OrbisGen2Result.ORBIS_GEN2_OK) : (int)OrbisGen2Result.ORBIS_GEN2_ERROR_BUSY; + if (result != (int)OrbisGen2Result.ORBIS_GEN2_ERROR_BUSY) + { + RestoreCondWaiterRecursion(waiter); + } + TracePthreadCond( waiter.CompletionState == 2 ? "wait-resume-timeout" : "wait-resume", condAddress, @@ -2465,6 +2523,14 @@ public static class KernelPthreadCompatExports return result; } + private static void RestoreCondWaiterRecursion(PthreadCondWaiter waiter) + { + if (waiter.SavedRecursion > 1 && waiter.MutexState.OwnerThreadId == waiter.ThreadId) + { + waiter.MutexState.RecursionCount = waiter.SavedRecursion; + } + } + private static int CondTimedOutResult(PthreadCondWaiter waiter) => waiter.PosixErrors ? 60 // ETIMEDOUT on Orbis/FreeBSD; pthread APIs return errno directly. diff --git a/src/SharpEmu.Libs/Kernel/KernelSyncOnAddressCompatExports.cs b/src/SharpEmu.Libs/Kernel/KernelSyncOnAddressCompatExports.cs index 17d81da..7dff0eb 100644 --- a/src/SharpEmu.Libs/Kernel/KernelSyncOnAddressCompatExports.cs +++ b/src/SharpEmu.Libs/Kernel/KernelSyncOnAddressCompatExports.cs @@ -144,7 +144,8 @@ public static class KernelSyncOnAddressCompatExports { if (Volatile.Read(ref _wakeRequested) == 0) { - Monitor.Wait(_hostGate, waitSlice); + // Round up so a sub-millisecond remainder does not spin. + Monitor.Wait(_hostGate, (int)Math.Ceiling(waitSlice.TotalMilliseconds)); } } diff --git a/src/SharpEmu.Libs/Media/HostMovieBridge.cs b/src/SharpEmu.Libs/Media/HostMovieBridge.cs index 27d7678..9873ee7 100644 --- a/src/SharpEmu.Libs/Media/HostMovieBridge.cs +++ b/src/SharpEmu.Libs/Media/HostMovieBridge.cs @@ -44,6 +44,8 @@ internal static class HostMovieBridge private static bool _frameBufferPresented; private static MediaFramePlayback? _playback; private static Timer? _playbackWatchdog; + private static long _watchdogFrameIndex = -1; + private static readonly TimeSpan StalledPlaybackTimeout = TimeSpan.FromSeconds(60); private static long _frameSerial; private static uint _presentationWidth = MaxHostVideoWidth; private static uint _presentationHeight = MaxHostVideoHeight; @@ -326,6 +328,7 @@ internal static class HostMovieBridge private static void ArmPlaybackWatchdogLocked(string hostPath) { _playbackWatchdog?.Dispose(); + _watchdogFrameIndex = -1; var timeout = TryReadBinkInfo(hostPath, out var info) ? GetPlaybackWatchdogTimeout(info) : TimeSpan.FromSeconds(150); @@ -340,6 +343,17 @@ internal static class HostMovieBridge return; } + // Playback follows the guest's own decode and audio clock, so a slow host plays a + // movie slower than real time. Expire only a movie that stopped advancing: forcing EOF + // under a guest decoder that is still mid-stream wedges it on the last frame. + var frameIndex = _playback?.PlaybackProgress.FrameIndex ?? -1; + if (_playback is not null && frameIndex > _watchdogFrameIndex) + { + _watchdogFrameIndex = frameIndex; + _playbackWatchdog?.Change(StalledPlaybackTimeout, Timeout.InfiniteTimeSpan); + return; + } + Console.Error.WriteLine( "[LOADER][WARN] Bink2 host watchdog expired for " + Path.GetFileName(path) + "; advancing to the next movie."); diff --git a/src/SharpEmu.Libs/Network/HttpExports.cs b/src/SharpEmu.Libs/Network/HttpExports.cs index 06f8a15..f6d0983 100644 --- a/src/SharpEmu.Libs/Network/HttpExports.cs +++ b/src/SharpEmu.Libs/Network/HttpExports.cs @@ -23,7 +23,9 @@ public static partial class HttpExports ulong UserAgentAddress, int HttpVersion, bool AutoProxyConfig, - uint ConnectTimeoutMicroseconds = 30_000_000); + uint ConnectTimeoutMicroseconds = 30_000_000, + uint ReceiveTimeoutMicroseconds = 120_000_000, + uint SendTimeoutMicroseconds = 120_000_000); [SysAbiExport( Nid = "A9cVMUtEp4Y", @@ -107,6 +109,72 @@ public static partial class HttpExports return ctx.SetReturn(0); } + [SysAbiExport( + Nid = "yigr4V0-HTM", + ExportName = "sceHttpSetRecvTimeOut", + Target = Generation.Gen4 | Generation.Gen5, + LibraryName = "libSceHttp")] + public static int HttpSetRecvTimeOut(CpuContext ctx) + { + var id = unchecked((int)ctx[CpuRegister.Rdi]); + var timeoutMicroseconds = unchecked((uint)ctx[CpuRegister.Rsi]); + if (timeoutMicroseconds == 0) + { + return ctx.SetReturn(HttpErrorInvalidValue); + } + + if (!Templates.TryGetValue(id, out var template)) + { + return ctx.SetReturn(HttpErrorInvalidId); + } + + Templates[id] = template with { ReceiveTimeoutMicroseconds = timeoutMicroseconds }; + TraceHttp("set_recv_timeout", id, timeoutMicroseconds, 0, 0, 0); + return ctx.SetReturn(0); + } + + [SysAbiExport( + Nid = "xegFfZKBVlw", + ExportName = "sceHttpSetSendTimeOut", + Target = Generation.Gen4 | Generation.Gen5, + LibraryName = "libSceHttp")] + public static int HttpSetSendTimeOut(CpuContext ctx) + { + var id = unchecked((int)ctx[CpuRegister.Rdi]); + var timeoutMicroseconds = unchecked((uint)ctx[CpuRegister.Rsi]); + if (timeoutMicroseconds == 0) + { + return ctx.SetReturn(HttpErrorInvalidValue); + } + + if (!Templates.TryGetValue(id, out var template)) + { + return ctx.SetReturn(HttpErrorInvalidId); + } + + Templates[id] = template with { SendTimeoutMicroseconds = timeoutMicroseconds }; + TraceHttp("set_send_timeout", id, timeoutMicroseconds, 0, 0, 0); + return ctx.SetReturn(0); + } + + // Registers CA certificates on an HTTP context; no host TLS session uses them. + [SysAbiExport( + Nid = "DK+GoXCNT04", + ExportName = "sceHttpsLoadCert", + Target = Generation.Gen4 | Generation.Gen5, + LibraryName = "libSceHttp")] + public static int HttpsLoadCert(CpuContext ctx) + { + var contextId = unchecked((int)ctx[CpuRegister.Rdi]); + if (!Contexts.ContainsKey(contextId)) + { + return ctx.SetReturn(HttpErrorInvalidId); + } + + TraceHttp("https_load_cert", contextId, ctx[CpuRegister.Rsi], ctx[CpuRegister.Rdx], ctx[CpuRegister.Rcx], ctx[CpuRegister.R8]); + return ctx.SetReturn(0); + } + [SysAbiExport( Nid = "Ik-KpLTlf7Q", ExportName = "sceHttpTerm", diff --git a/src/SharpEmu.Libs/Network/NetExports.cs b/src/SharpEmu.Libs/Network/NetExports.cs index 0f6938b..7a869e9 100644 --- a/src/SharpEmu.Libs/Network/NetExports.cs +++ b/src/SharpEmu.Libs/Network/NetExports.cs @@ -14,42 +14,30 @@ namespace SharpEmu.Libs.Network; public static class NetExports { - private const int NetErrorNoEntry = unchecked((int)0x80410102); private const int NetErrorBadFileDescriptor = unchecked((int)0x80410109); private const int NetErrorFault = unchecked((int)0x8041010E); - private const int NetErrorAlreadyExists = unchecked((int)0x80410111); private const int NetErrorInvalidArgument = unchecked((int)0x80410116); private const int NetErrorTooManyFiles = unchecked((int)0x80410118); private const int NetErrorWouldBlock = unchecked((int)0x80410123); private const int NetErrorAddressInUse = unchecked((int)0x80410130); - private const int NetErrorTooManyOpenFiles = unchecked((int)0x80410118); private const int NetErrorResolverNoDns = unchecked((int)0x804101E1); private const int NetErrorNotInitialized = unchecked((int)0x804101C8); - private const int NetErrnoNoEntry = 2; private const int NetErrnoBadFileDescriptor = 9; private const int NetErrnoFault = 14; - private const int NetErrnoAlreadyExists = 17; private const int NetErrnoInvalidArgument = 22; private const int NetErrnoTooManyFiles = 24; private const int NetErrnoWouldBlock = 35; private const int NetErrnoAddressInUse = 48; - private const int NetErrnoTooManyOpenFiles = 24; private const int NetErrnoNotInitialized = 200; private const int NetErrnoResolverNoDns = 225; private const int MaxNameLength = 256; - private const int EpollIdBase = 1024; - private const int EpollCapacity = 32; - private const int NetEpollEventSize = 24; - private const uint NetEpollIn = 0x00000001; - private const uint NetEpollOut = 0x00000002; - private const uint NetEpollError = 0x00000008; private static ReadOnlySpan OfflineMacAddress => [0x02, 0x53, 0x48, 0x41, 0x52, 0x50]; private static readonly ConcurrentDictionary _pools = new(); private static readonly ConcurrentDictionary _resolvers = new(); private static readonly ConcurrentDictionary _sockets = new(); - private static readonly object EpollGate = new(); - private static readonly Dictionary _epolls = new(); + private static readonly ConcurrentDictionary _epolls = new(); + private static int _nextEpollId = 0x6000; private static int _nextPoolId; private static int _nextResolverId = 0x2000; private static int _nextSocketId = 255; @@ -64,28 +52,178 @@ public static class NetExports private sealed record NetPool(string Name, int Size, int Flags); - private sealed record ResolverContext(string Name, int PoolId, int Flags, int LastError); - - private sealed class NetEpollContext(string name) + private sealed class NetEpoll(string name) { public string Name { get; } = name; - - public List Registrations { get; } = []; + public object Gate { get; } = new(); + public Dictionary Watches { get; } = new(); + public bool Destroyed; } - private readonly record struct NetEpollRegistration(int SocketId, NetEpollEvent Event); - - private readonly record struct NetEpollEvent(uint Events, ulong Data); - - internal static void ResetEpollsForTests() + [SysAbiExport( + Nid = "SF47kB2MNTo", + ExportName = "sceNetEpollCreate", + Target = Generation.Gen4 | Generation.Gen5, + LibraryName = "libSceNet")] + public static int NetEpollCreate(CpuContext ctx) { - lock (EpollGate) + if (!_initialized) + return SetNetError(ctx, NetErrorNotInitialized, NetErrnoNotInitialized); + if (ctx[CpuRegister.Rsi] != 0) + return SetNetError(ctx, NetErrorInvalidArgument, NetErrnoInvalidArgument); + + var name = string.Empty; + if (ctx[CpuRegister.Rdi] != 0 && + !TryReadUtf8Z(ctx, ctx[CpuRegister.Rdi], MaxNameLength, out name)) + return SetNetError(ctx, NetErrorFault, NetErrnoFault); + + var id = Interlocked.Increment(ref _nextEpollId); + if (id <= 0 || !_epolls.TryAdd(id, new NetEpoll(name))) + return SetNetError(ctx, NetErrorTooManyFiles, NetErrnoTooManyFiles); + ctx[CpuRegister.Rax] = unchecked((ulong)id); + return 0; + } + + [SysAbiExport( + Nid = "Inp1lfL+Jdw", + ExportName = "sceNetEpollDestroy", + Target = Generation.Gen4 | Generation.Gen5, + LibraryName = "libSceNet")] + public static int NetEpollDestroy(CpuContext ctx) + { + if (!_epolls.TryRemove(unchecked((int)ctx[CpuRegister.Rdi]), out var poll)) + return SetNetError(ctx, NetErrorBadFileDescriptor, NetErrnoBadFileDescriptor); + DestroyEpoll(poll); + return ctx.SetReturn(0); + } + + private static void DestroyEpoll(NetEpoll poll) + { + lock (poll.Gate) { - _epolls.Clear(); - Monitor.PulseAll(EpollGate); + poll.Destroyed = true; + poll.Watches.Clear(); + Monitor.PulseAll(poll.Gate); } } + [SysAbiExport(Nid = "ZVw46bsasAk", ExportName = "sceNetEpollControl", + Target = Generation.Gen4 | Generation.Gen5, LibraryName = "libSceNet")] + public static int NetEpollControl(CpuContext ctx) + { + if (!_epolls.TryGetValue(unchecked((int)ctx[CpuRegister.Rdi]), out var poll)) + return SetNetError(ctx, NetErrorBadFileDescriptor, NetErrnoBadFileDescriptor); + var operation = unchecked((int)ctx[CpuRegister.Rsi]); + var socketId = unchecked((int)ctx[CpuRegister.Rdx]); + var eventAddress = ctx[CpuRegister.Rcx]; + if (operation is < 1 or > 3 || (operation == 3 && eventAddress != 0)) + return SetNetError(ctx, NetErrorInvalidArgument, NetErrnoInvalidArgument); + if (!_sockets.ContainsKey(socketId)) + return SetNetError(ctx, NetErrorBadFileDescriptor, NetErrnoBadFileDescriptor); + uint events = 0; + ulong data = 0; + if (operation != 3) + { + Span record = stackalloc byte[24]; + if (eventAddress == 0) + return SetNetError(ctx, NetErrorInvalidArgument, NetErrnoInvalidArgument); + if (!ctx.Memory.TryRead(eventAddress, record)) + return SetNetError(ctx, NetErrorFault, NetErrnoFault); + events = BinaryPrimitives.ReadUInt32LittleEndian(record); + data = BinaryPrimitives.ReadUInt64LittleEndian(record[16..]); + // Only level-triggered socket readiness is supported. Reject edge, + // one-shot and resolver registrations instead of changing their meaning. + if ((events & ~0x1001Bu) != 0) + return SetNetError(ctx, NetErrorInvalidArgument, NetErrnoInvalidArgument); + } + lock (poll.Gate) + { + if (poll.Destroyed) + return SetNetError(ctx, NetErrorBadFileDescriptor, NetErrnoBadFileDescriptor); + if (operation == 1 && poll.Watches.ContainsKey(socketId)) + return SetNetError(ctx, unchecked((int)0x80410111), 17); // EEXIST + if (operation != 1 && !poll.Watches.ContainsKey(socketId)) + return SetNetError(ctx, NetErrorBadFileDescriptor, NetErrnoBadFileDescriptor); + if (operation == 3) poll.Watches.Remove(socketId); + else poll.Watches[socketId] = (events, data); + Monitor.PulseAll(poll.Gate); + } + return ctx.SetReturn(0); + } + + [SysAbiExport(Nid = "drjIbDbA7UQ", ExportName = "sceNetEpollWait", + Target = Generation.Gen4 | Generation.Gen5, LibraryName = "libSceNet")] + public static int NetEpollWait(CpuContext ctx) + { + if (!_epolls.TryGetValue(unchecked((int)ctx[CpuRegister.Rdi]), out var poll)) + return SetNetError(ctx, NetErrorBadFileDescriptor, NetErrnoBadFileDescriptor); + var output = ctx[CpuRegister.Rsi]; + var capacity = unchecked((int)ctx[CpuRegister.Rdx]); + var timeoutUsec = unchecked((int)ctx[CpuRegister.Rcx]); + if (capacity <= 0) + return SetNetError(ctx, NetErrorInvalidArgument, NetErrnoInvalidArgument); + if (output == 0 || output > ulong.MaxValue - (ulong)capacity * 24) + return SetNetError(ctx, NetErrorFault, NetErrnoFault); + var deadline = timeoutUsec < 0 ? long.MaxValue : + GuestThreadExecution.ComputeDeadlineTimestamp(TimeSpan.FromTicks((long)timeoutUsec * 10)); + Span record = stackalloc byte[24]; + while (true) + { + var count = 0; + lock (poll.Gate) + { + if (poll.Destroyed) + return SetNetError(ctx, NetErrorBadFileDescriptor, NetErrnoBadFileDescriptor); + foreach (var (id, watch) in poll.Watches) + { + if (!_sockets.TryGetValue(id, out var socket)) continue; + uint ready = 0; + try + { + if (socket.Poll(0, SelectMode.SelectRead)) + { + ready |= watch.Events & 1; + // A pending accept also signals readability without payload bytes. + if (socket.SocketType == SocketType.Stream && + (int)socket.GetSocketOption(SocketOptionLevel.Socket, + SocketOptionName.AcceptConnection)! == 0 && socket.Available == 0) + ready |= 0x10; + } + if ((watch.Events & 2) != 0 && socket.Poll(0, SelectMode.SelectWrite)) ready |= 2; + if (socket.Poll(0, SelectMode.SelectError) && + (int)socket.GetSocketOption(SocketOptionLevel.Socket, SocketOptionName.Error)! != 0) + ready |= 8; + } + catch (ObjectDisposedException) { continue; } + catch (SocketException) { ready |= 8; } + if (ready == 0) continue; + record.Clear(); + BinaryPrimitives.WriteUInt32LittleEndian(record, ready); + BinaryPrimitives.WriteUInt64LittleEndian(record[8..], unchecked((ulong)id)); + BinaryPrimitives.WriteUInt64LittleEndian(record[16..], watch.Data); + if (!ctx.Memory.TryWrite(output + (ulong)count * 24, record)) + return SetNetError(ctx, NetErrorFault, NetErrnoFault); + if (++count == capacity) break; + } + if (count != 0 || System.Diagnostics.Stopwatch.GetTimestamp() >= deadline) + { + ctx[CpuRegister.Rax] = unchecked((ulong)count); + return 0; + } + // Socket activity is checked at bounded intervals; control/destroy + // wakes this monitor immediately. No event is invented on timeout. + var remaining = deadline == long.MaxValue ? TimeSpan.FromMilliseconds(1) : + TimeSpan.FromSeconds((deadline - System.Diagnostics.Stopwatch.GetTimestamp()) / + (double)System.Diagnostics.Stopwatch.Frequency); + if (remaining > TimeSpan.Zero) + Monitor.Wait(poll.Gate, remaining < TimeSpan.FromMilliseconds(1) ? remaining : TimeSpan.FromMilliseconds(1)); + } + GuestThreadExecution.Scheduler?.DeliverPendingGuestExceptionIfReady(ctx); + } + } + + private sealed record ResolverContext(string Name, int PoolId, int Flags, int LastError); + internal static bool TryGetReadEventState( int socketId, ulong lowWater, @@ -150,11 +288,8 @@ public static class NetExports _initialized = false; _pools.Clear(); _resolvers.Clear(); - lock (EpollGate) - { - _epolls.Clear(); - Monitor.PulseAll(EpollGate); - } + foreach (var poll in _epolls.Values) DestroyEpoll(poll); + _epolls.Clear(); foreach (var socket in _sockets.Values) { socket.Dispose(); @@ -182,6 +317,36 @@ public static class NetExports return ctx.SetReturn(0); } + [SysAbiExport( + Nid = "v6M4txecCuo", + ExportName = "sceNetEtherNtostr", + Target = Generation.Gen4 | Generation.Gen5, + LibraryName = "libSceNet")] + public static int NetEtherNtostr(CpuContext ctx) + { + var address = ctx[CpuRegister.Rdi]; + var outputAddress = ctx[CpuRegister.Rsi]; + var capacity = unchecked((int)ctx[CpuRegister.Rdx]); + if (address == 0 || outputAddress == 0 || capacity < 18) + return SetNetError(ctx, NetErrorInvalidArgument, NetErrnoInvalidArgument); + + Span mac = stackalloc byte[6]; + if (!ctx.Memory.TryRead(address, mac)) + return SetNetError(ctx, NetErrorFault, NetErrnoFault); + + const string hex = "0123456789abcdef"; + Span text = stackalloc byte[18]; + for (var index = 0; index < mac.Length; index++) + { + text[index * 3] = (byte)hex[mac[index] >> 4]; + text[index * 3 + 1] = (byte)hex[mac[index] & 15]; + text[index * 3 + 2] = index == 5 ? (byte)0 : (byte)':'; + } + if (!ctx.Memory.TryWrite(outputAddress, text)) + return SetNetError(ctx, NetErrorFault, NetErrnoFault); + return ctx.SetReturn(0); + } + [SysAbiExport( Nid = "Q4qBuN-c0ZM", ExportName = "sceNetSocket", @@ -234,12 +399,6 @@ public static class NetExports public static int NetSocketClose(CpuContext ctx) { var id = unchecked((int)ctx[CpuRegister.Rdi]); - if (KernelSocketCompatExports.TryCloseSocketFd(id)) - { - TraceNet("socket.close.posix", id, 0, 0, 0); - return ctx.SetReturn(0); - } - if (!_sockets.TryRemove(id, out var socket)) { return SetNetError(ctx, NetErrorBadFileDescriptor, NetErrnoBadFileDescriptor); @@ -760,291 +919,6 @@ public static class NetExports return (int)OrbisGen2Result.ORBIS_GEN2_OK; } - [SysAbiExport( - Nid = "SF47kB2MNTo", - ExportName = "sceNetEpollCreate", - Target = Generation.Gen4 | Generation.Gen5, - LibraryName = "libSceNet")] - public static int NetEpollCreate(CpuContext ctx) - { - if (!_initialized) - { - return SetNetError(ctx, NetErrorNotInitialized, NetErrnoNotInitialized); - } - - var nameAddress = ctx[CpuRegister.Rdi]; - var flags = unchecked((int)ctx[CpuRegister.Rsi]); - if (nameAddress == 0 || flags != 0 || - !TryReadUtf8Z(ctx, nameAddress, MaxNameLength, out var name)) - { - return SetNetError(ctx, NetErrorInvalidArgument, NetErrnoInvalidArgument); - } - - lock (EpollGate) - { - for (var index = 0; index < EpollCapacity; index++) - { - var id = EpollIdBase + index; - if (_epolls.ContainsKey(id)) - { - continue; - } - - _epolls.Add(id, new NetEpollContext(name)); - TraceNet("epoll.create", id, nameAddress, 0, 0); - return ctx.SetReturn(id); - } - } - - return SetNetError(ctx, NetErrorTooManyOpenFiles, NetErrnoTooManyOpenFiles); - } - - [SysAbiExport( - Nid = "ZVw46bsasAk", - ExportName = "sceNetEpollControl", - Target = Generation.Gen4 | Generation.Gen5, - LibraryName = "libSceNet")] - public static int NetEpollControl(CpuContext ctx) - { - const int EpollControlAdd = 1; - const int EpollControlModify = 2; - const int EpollControlDelete = 3; - - var epollId = unchecked((int)ctx[CpuRegister.Rdi]); - var operation = unchecked((int)ctx[CpuRegister.Rsi]); - var socketId = unchecked((int)ctx[CpuRegister.Rdx]); - var eventAddress = ctx[CpuRegister.Rcx]; - - if (operation is < EpollControlAdd or > EpollControlDelete || - (operation is EpollControlAdd or EpollControlModify && eventAddress == 0) || - (operation == EpollControlDelete && eventAddress != 0)) - { - return SetNetError(ctx, NetErrorInvalidArgument, NetErrnoInvalidArgument); - } - - if (!_sockets.ContainsKey(socketId)) - { - return SetNetError(ctx, NetErrorBadFileDescriptor, NetErrnoBadFileDescriptor); - } - - NetEpollEvent requestedEvent = default; - if (eventAddress != 0 && !TryReadNetEpollEvent(ctx, eventAddress, out requestedEvent)) - { - return SetNetError(ctx, NetErrorFault, NetErrnoFault); - } - - lock (EpollGate) - { - if (!_epolls.TryGetValue(epollId, out var epoll)) - { - return SetNetError(ctx, NetErrorBadFileDescriptor, NetErrnoBadFileDescriptor); - } - - var registrationIndex = epoll.Registrations.FindIndex(value => value.SocketId == socketId); - switch (operation) - { - case EpollControlAdd when registrationIndex >= 0: - return SetNetError(ctx, NetErrorAlreadyExists, NetErrnoAlreadyExists); - case EpollControlAdd: - epoll.Registrations.Add(new NetEpollRegistration(socketId, requestedEvent)); - break; - case EpollControlModify when registrationIndex < 0: - return SetNetError(ctx, NetErrorNoEntry, NetErrnoNoEntry); - case EpollControlModify: - epoll.Registrations[registrationIndex] = - new NetEpollRegistration(socketId, requestedEvent); - break; - case EpollControlDelete when registrationIndex < 0: - return SetNetError(ctx, NetErrorNoEntry, NetErrnoNoEntry); - case EpollControlDelete: - epoll.Registrations.RemoveAt(registrationIndex); - break; - } - - Monitor.PulseAll(EpollGate); - } - - TraceNet( - "epoll.control", - epollId, - unchecked((uint)operation), - unchecked((uint)socketId), - requestedEvent.Events); - return ctx.SetReturn(0); - } - - [SysAbiExport( - Nid = "drjIbDbA7UQ", - ExportName = "sceNetEpollWait", - Target = Generation.Gen4 | Generation.Gen5, - LibraryName = "libSceNet")] - public static int NetEpollWait(CpuContext ctx) - { - var epollId = unchecked((int)ctx[CpuRegister.Rdi]); - var eventsAddress = ctx[CpuRegister.Rsi]; - var maxEvents = unchecked((int)ctx[CpuRegister.Rdx]); - var timeoutMicroseconds = unchecked((int)ctx[CpuRegister.Rcx]); - - if (eventsAddress == 0) - { - return SetNetError(ctx, NetErrorFault, NetErrnoFault); - } - - if (maxEvents <= 0 || timeoutMicroseconds < -1) - { - return SetNetError(ctx, NetErrorInvalidArgument, NetErrnoInvalidArgument); - } - - NetEpollRegistration[] registrations; - lock (EpollGate) - { - if (!_epolls.TryGetValue(epollId, out var epoll)) - { - return SetNetError(ctx, NetErrorBadFileDescriptor, NetErrnoBadFileDescriptor); - } - - if (epoll.Registrations.Count == 0 && timeoutMicroseconds != 0) - { - if (timeoutMicroseconds < 0) - { - while (_epolls.TryGetValue(epollId, out epoll) && epoll.Registrations.Count == 0) - { - Monitor.Wait(EpollGate); - } - } - else - { - Monitor.Wait( - EpollGate, - TimeSpan.FromTicks((long)timeoutMicroseconds * TimeSpan.TicksPerMicrosecond)); - } - - if (!_epolls.TryGetValue(epollId, out epoll)) - { - return SetNetError(ctx, NetErrorBadFileDescriptor, NetErrnoBadFileDescriptor); - } - } - - registrations = epoll.Registrations.ToArray(); - } - - if (registrations.Length == 0) - { - TraceNet("epoll.wait", epollId, unchecked((uint)maxEvents), unchecked((uint)timeoutMicroseconds), 0); - return ctx.SetReturn(0); - } - - var readSockets = new List(); - var writeSockets = new List(); - var errorSockets = new List(); - var hostRegistrations = new Dictionary(); - foreach (var registration in registrations) - { - if (!_sockets.TryGetValue(registration.SocketId, out var socket)) - { - continue; - } - - hostRegistrations[socket] = registration; - if ((registration.Event.Events & NetEpollIn) != 0) - { - readSockets.Add(socket); - } - if ((registration.Event.Events & NetEpollOut) != 0) - { - writeSockets.Add(socket); - } - errorSockets.Add(socket); - } - - if (hostRegistrations.Count == 0) - { - return ctx.SetReturn(0); - } - - try - { - Socket.Select(readSockets, writeSockets, errorSockets, timeoutMicroseconds); - } - catch (ObjectDisposedException) - { - return SetNetError(ctx, NetErrorBadFileDescriptor, NetErrnoBadFileDescriptor); - } - catch (SocketException) - { - return SetNetError(ctx, NetErrorInvalidArgument, NetErrnoInvalidArgument); - } - - var readyCount = 0; - foreach (var (socket, registration) in hostRegistrations) - { - uint readyEvents = 0; - if (readSockets.Contains(socket)) - { - readyEvents |= NetEpollIn; - } - if (writeSockets.Contains(socket)) - { - readyEvents |= NetEpollOut; - } - if (errorSockets.Contains(socket)) - { - readyEvents |= NetEpollError; - } - if (readyEvents == 0) - { - continue; - } - - var outputAddress = eventsAddress + (ulong)(readyCount * NetEpollEventSize); - if (!TryWriteNetEpollEvent( - ctx, - outputAddress, - readyEvents, - registration.SocketId, - registration.Event.Data)) - { - return SetNetError(ctx, NetErrorFault, NetErrnoFault); - } - - readyCount++; - if (readyCount == maxEvents) - { - break; - } - } - - TraceNet( - "epoll.wait", - epollId, - unchecked((uint)maxEvents), - unchecked((uint)timeoutMicroseconds), - unchecked((uint)readyCount)); - return ctx.SetReturn(readyCount); - } - - [SysAbiExport( - Nid = "Inp1lfL+Jdw", - ExportName = "sceNetEpollDestroy", - Target = Generation.Gen4 | Generation.Gen5, - LibraryName = "libSceNet")] - public static int NetEpollDestroy(CpuContext ctx) - { - var epollId = unchecked((int)ctx[CpuRegister.Rdi]); - lock (EpollGate) - { - if (!_epolls.Remove(epollId)) - { - return SetNetError(ctx, NetErrorBadFileDescriptor, NetErrnoBadFileDescriptor); - } - - Monitor.PulseAll(EpollGate); - } - - TraceNet("epoll.destroy", epollId, 0, 0, 0); - return ctx.SetReturn(0); - } - [SysAbiExport( Nid = "dgJBaeJnGpo", ExportName = "sceNetPoolCreate", @@ -1146,53 +1020,6 @@ public static class NetExports return (int)OrbisGen2Result.ORBIS_GEN2_OK; } - [SysAbiExport( - Nid = "v6M4txecCuo", - ExportName = "sceNetEtherNtostr", - Target = Generation.Gen4 | Generation.Gen5, - LibraryName = "libSceNet")] - public static int NetEtherNtostr(CpuContext ctx) - { - const int EtherStringLength = 18; - const string HexDigits = "0123456789abcdef"; - - var address = ctx[CpuRegister.Rdi]; - var destination = ctx[CpuRegister.Rsi]; - var destinationLength = ctx[CpuRegister.Rdx]; - if (address == 0 || destination == 0 || destinationLength < EtherStringLength) - { - return SetNetError(ctx, NetErrorInvalidArgument, NetErrnoInvalidArgument); - } - - Span macAddress = stackalloc byte[6]; - if (!ctx.Memory.TryRead(address, macAddress)) - { - return SetNetError(ctx, NetErrorInvalidArgument, NetErrnoInvalidArgument); - } - - Span text = stackalloc byte[EtherStringLength]; - for (var index = 0; index < macAddress.Length; index++) - { - var value = macAddress[index]; - var offset = index * 3; - text[offset] = unchecked((byte)HexDigits[value >> 4]); - text[offset + 1] = unchecked((byte)HexDigits[value & 0xF]); - if (index != macAddress.Length - 1) - { - text[offset + 2] = (byte)':'; - } - } - text[^1] = 0; - - if (!ctx.Memory.TryWrite(destination, text)) - { - return SetNetError(ctx, NetErrorInvalidArgument, NetErrnoInvalidArgument); - } - - TraceNet("ether_ntostr", 0, address, destination, destinationLength); - return ctx.SetReturn(0); - } - [SysAbiExport( Nid = "C4UgDHHPvdw", ExportName = "sceNetResolverCreate", @@ -1402,39 +1229,6 @@ public static class NetExports return true; } - private static bool TryReadNetEpollEvent( - CpuContext ctx, - ulong address, - out NetEpollEvent epollEvent) - { - Span bytes = stackalloc byte[NetEpollEventSize]; - if (!ctx.Memory.TryRead(address, bytes)) - { - epollEvent = default; - return false; - } - - epollEvent = new NetEpollEvent( - BinaryPrimitives.ReadUInt32LittleEndian(bytes), - BinaryPrimitives.ReadUInt64LittleEndian(bytes[16..])); - return true; - } - - private static bool TryWriteNetEpollEvent( - CpuContext ctx, - ulong address, - uint events, - int socketId, - ulong data) - { - Span bytes = stackalloc byte[NetEpollEventSize]; - bytes.Clear(); - BinaryPrimitives.WriteUInt32LittleEndian(bytes, events); - BinaryPrimitives.WriteUInt64LittleEndian(bytes[8..], unchecked((uint)socketId)); - BinaryPrimitives.WriteUInt64LittleEndian(bytes[16..], data); - return ctx.Memory.TryWrite(address, bytes); - } - [SysAbiExport( Nid = "8Kcp5d-q1Uo", ExportName = "sceNetInetPton", diff --git a/src/SharpEmu.Libs/Np/NpTrophy2Exports.cs b/src/SharpEmu.Libs/Np/NpTrophy2Exports.cs index 4fe0f43..0a943e8 100644 --- a/src/SharpEmu.Libs/Np/NpTrophy2Exports.cs +++ b/src/SharpEmu.Libs/Np/NpTrophy2Exports.cs @@ -104,8 +104,217 @@ public static class NpTrophy2Exports ExportName = "sceNpTrophy2GetTrophyInfoArray", Target = Generation.Gen5, LibraryName = "libSceNpTrophy2")] - public static int NpTrophy2GetTrophyInfoArray(CpuContext ctx) => - SetReturn(ctx, OrbisGen2Result.ORBIS_GEN2_ERROR_NOT_FOUND); + public static int NpTrophy2GetTrophyInfoArray(CpuContext ctx) + { + var offset = (uint)ctx[CpuRegister.Rdx]; + var limit = (uint)ctx[CpuRegister.Rcx]; + var detailsAddress = ctx[CpuRegister.R8]; + var dataAddress = ctx[CpuRegister.R9]; + if ((detailsAddress == 0 && dataAddress == 0) || + !ctx.TryGetImportStackArgument(0, out var countAddress) || countAddress == 0) + { + return SetReturn(ctx, OrbisGen2Result.ORBIS_GEN2_ERROR_INVALID_ARGUMENT); + } + + // The count is how many trophies the package really holds from offset on, capped by + // limit. Reporting limit itself makes a title walk past its own arrays. + var trophies = LoadTrophies(); + var start = (int)Math.Min(offset, (uint)trophies.Length); + var count = (int)Math.Min(limit, (uint)(trophies.Length - start)); + Span details = stackalloc byte[TrophyDetailsSize]; + Span data = stackalloc byte[TrophyDataSize]; + for (var index = 0; index < count; index++) + { + var trophy = trophies[start + index]; + if (detailsAddress != 0) + { + trophy.WriteDetails(details); + if (!ctx.Memory.TryWrite(detailsAddress + (ulong)index * TrophyDetailsSize, details)) + { + return SetReturn(ctx, OrbisGen2Result.ORBIS_GEN2_ERROR_MEMORY_FAULT); + } + } + + if (dataAddress != 0) + { + // Nothing is unlocked for the emulated user. + data.Clear(); + BinaryPrimitives.WriteInt32LittleEndian(data, trophy.Id); + if (!ctx.Memory.TryWrite(dataAddress + (ulong)index * TrophyDataSize, data)) + { + return SetReturn(ctx, OrbisGen2Result.ORBIS_GEN2_ERROR_MEMORY_FAULT); + } + } + } + + Span countBytes = stackalloc byte[sizeof(uint)]; + BinaryPrimitives.WriteUInt32LittleEndian(countBytes, (uint)count); + return ctx.Memory.TryWrite(countAddress, countBytes) + ? SetReturn(ctx, OrbisGen2Result.ORBIS_GEN2_OK) + : SetReturn(ctx, OrbisGen2Result.ORBIS_GEN2_ERROR_MEMORY_FAULT); + } + + // SceNpTrophy2Details is 1312 bytes: id/grade/group at +0/+4/+8, hidden at +12, has-reward at + // +13, progress target (type, value) at +16, then name[128], description[1024], reward[128]. + // SceNpTrophy2Data is 32 bytes: id at +0, unlocked flag at +4, progress at +8, timestamp at +24. + private const int TrophyDetailsSize = 1312; + private const int TrophyDataSize = 32; + + private static readonly object TrophyGate = new(); + private static TrophyInfo[]? _trophies; + + private sealed record TrophyInfo( + int Id, int Grade, int GroupId, bool Hidden, bool HasReward, string Name, string Description) + { + public void WriteDetails(Span destination) + { + destination.Clear(); + BinaryPrimitives.WriteInt32LittleEndian(destination, Id); + BinaryPrimitives.WriteInt32LittleEndian(destination[4..], Grade); + BinaryPrimitives.WriteInt32LittleEndian(destination[8..], GroupId); + destination[12] = Hidden ? (byte)1 : (byte)0; + destination[13] = HasReward ? (byte)1 : (byte)0; + WriteText(destination.Slice(32, 128), Name); + WriteText(destination.Slice(160, 1024), Description); + } + + private static void WriteText(Span field, string text) + { + var bytes = System.Text.Encoding.UTF8.GetBytes(text); + var length = Math.Min(bytes.Length, field.Length - 1); + bytes.AsSpan(0, length).CopyTo(field); + } + } + + private static TrophyInfo[] LoadTrophies() + { + lock (TrophyGate) + { + return _trophies ??= ReadTrophyPackage(); + } + } + + // sce_sys/trophy2/trophy00.ucp is an unencrypted container: a big-endian header, a table of + // 0x40-byte entries (name[0x20], offset, length), and JSON files for the trophy list and the + // localized texts. + private static TrophyInfo[] ReadTrophyPackage() + { + try + { + var path = Kernel.KernelMemoryCompatExports.ResolveGuestPath("/app0/sce_sys/trophy2/trophy00.ucp"); + if (!File.Exists(path)) + { + return []; + } + + var package = File.ReadAllBytes(path); + if (package.Length < 0x40 || + BinaryPrimitives.ReadUInt32BigEndian(package) != 0xB228C60A) + { + return []; + } + + var fileCount = BinaryPrimitives.ReadUInt32BigEndian(package.AsSpan(0x10)); + var tableOffset = BinaryPrimitives.ReadUInt32BigEndian(package.AsSpan(0x14)); + var files = new Dictionary(); + for (var i = 0; i < Math.Min(fileCount, 4096u); i++) + { + var entry = (long)tableOffset + 0x20 + i * 0x40L; + if (entry + 0x40 > package.Length) + { + break; + } + + var nameBytes = package.AsSpan((int)entry, 0x20); + var nameLength = nameBytes.IndexOf((byte)0); + var name = System.Text.Encoding.UTF8.GetString(nameBytes[..(nameLength < 0 ? nameBytes.Length : nameLength)]); + var offset = (long)BinaryPrimitives.ReadUInt64BigEndian(package.AsSpan((int)entry + 0x20)); + var length = (long)BinaryPrimitives.ReadUInt64BigEndian(package.AsSpan((int)entry + 0x28)); + if (offset >= 0 && length >= 0 && offset + length <= package.Length) + { + files[name] = (offset, length); + } + } + + System.Text.Json.JsonDocument? Json(string name) => + files.TryGetValue(name, out var file) + ? System.Text.Json.JsonDocument.Parse(package.AsMemory((int)file.Offset, (int)file.Length)) + : null; + + using var conf = Json("tropconf.json"); + if (conf is null || !conf.RootElement.TryGetProperty("trophies", out var list)) + { + return []; + } + + var language = conf.RootElement.TryGetProperty("defaultLanguage", out var lang) + ? lang.GetString() ?? "en-US" + : "en-US"; + using var meta = Json("tropmeta_en-US.json") ?? Json($"tropmeta_{language}.json"); + var texts = new Dictionary(); + if (meta is not null && + meta.RootElement.TryGetProperty("metadata", out var metadata) && + metadata.TryGetProperty("trophyMetadata", out var trophyTexts)) + { + foreach (var text in trophyTexts.EnumerateArray()) + { + if (TryReadId(text, "id", out var id)) + { + texts[id] = ( + text.TryGetProperty("name", out var n) ? n.GetString() ?? "" : "", + text.TryGetProperty("detail", out var d) ? d.GetString() ?? "" : ""); + } + } + } + + var result = new SortedDictionary(); + foreach (var definition in list.EnumerateArray()) + { + if (!TryReadId(definition, "id", out var id) || + !definition.TryGetProperty("grade", out var gradeElement) || + gradeElement.GetString() is not { Length: 1 } grade) + { + continue; + } + + var gradeIndex = "PGSB".IndexOf(grade[0]); + if (gradeIndex < 0) + { + continue; + } + + var groupId = TryReadId(definition, "groupId", out var group) ? group : 0; + texts.TryGetValue(id, out var trophyText); + result[id] = new TrophyInfo( + id, + gradeIndex + 1, + groupId, + definition.TryGetProperty("hidden", out var hidden) && hidden.ValueKind == System.Text.Json.JsonValueKind.True, + definition.TryGetProperty("hasReward", out var reward) && reward.ValueKind == System.Text.Json.JsonValueKind.True, + trophyText.Name ?? "", + trophyText.Detail ?? ""); + } + + return [.. result.Values]; + } + catch (Exception ex) when (ex is IOException or System.Text.Json.JsonException or InvalidOperationException or ArgumentException) + { + return []; + } + } + + private static bool TryReadId(System.Text.Json.JsonElement element, string property, out int id) + { + id = 0; + if (!element.TryGetProperty(property, out var value)) + { + return false; + } + + return value.ValueKind == System.Text.Json.JsonValueKind.Number + ? value.TryGetInt32(out id) + : int.TryParse(value.GetString(), out id); + } private static int WriteIdAndReturn(CpuContext ctx, ulong outAddress, ref int nextId) diff --git a/src/SharpEmu.Libs/Pad/TouchContactIds.cs b/src/SharpEmu.Libs/Pad/TouchContactIds.cs new file mode 100644 index 0000000..cb16a8d --- /dev/null +++ b/src/SharpEmu.Libs/Pad/TouchContactIds.cs @@ -0,0 +1,36 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +namespace SharpEmu.Libs.Pad; + +// The pad gives each new finger contact its own touch id, 1 to 127 and then back to 1, +// and keeps it while the finger stays down. Games tell strokes apart by that id: with a +// fixed id per finger slot, lifting the finger and touching elsewhere reads as one +// contact jumping across the pad (Ghost of Yōtei's name drawing never saw a new stroke). +internal sealed class TouchContactIds +{ + private const byte FirstId = 1; + private const byte LastId = 127; + + private readonly bool[] _down = new bool[2]; + private readonly byte[] _ids = new byte[2]; + private byte _next = FirstId; + + // The id to report for this finger slot; a contact that just started gets a new one. + public byte Track(int finger, bool down) + { + if ((uint)finger >= (uint)_down.Length) + { + return 0; + } + + if (down && !_down[finger]) + { + _ids[finger] = _next; + _next = _next == LastId ? FirstId : (byte)(_next + 1); + } + + _down[finger] = down; + return _ids[finger]; + } +} diff --git a/src/SharpEmu.Libs/PlayGo/PlayGoExports.cs b/src/SharpEmu.Libs/PlayGo/PlayGoExports.cs index 12ca714..0b16673 100644 --- a/src/SharpEmu.Libs/PlayGo/PlayGoExports.cs +++ b/src/SharpEmu.Libs/PlayGo/PlayGoExports.cs @@ -19,6 +19,7 @@ public static class PlayGoExports private const int OrbisPlayGoErrorBadChunkId = unchecked((int)0x80B2000C); private const int OrbisPlayGoErrorNotSupportPlayGo = unchecked((int)0x80B2000E); private const int OrbisPlayGoErrorBadLocus = unchecked((int)0x80B20010); + private const int OrbisKernelErrorNoEntry = unchecked((int)0x80020002); private const int OrbisPlayGoErrorBadOptionalType = unchecked((int)0x80B20024); private const ulong PlayGoInitBufAddrOffset = 0; private const ulong PlayGoInitBufSizeOffset = 8; @@ -254,8 +255,7 @@ public static class PlayGoExports for (uint i = 0; i < entriesToWrite; i++) { - var chunkId = chunkIds[i]; - if (!ctx.TryWriteUInt16(outChunkIdList + (i * sizeof(ushort)), chunkId)) + if (!ctx.TryWriteUInt16(outChunkIdList + (i * sizeof(ushort)), chunkIds[i])) { return (int)OrbisGen2Result.ORBIS_GEN2_ERROR_MEMORY_FAULT; } @@ -566,8 +566,8 @@ public static class PlayGoExports TracePlayGo($"get_progress entries={numberOfEntries}"); Span progress = stackalloc byte[sizeof(ulong) * 2]; - BinaryPrimitives.WriteUInt64LittleEndian(progress, 0); - BinaryPrimitives.WriteUInt64LittleEndian(progress[sizeof(ulong)..], 0); + BinaryPrimitives.WriteUInt64LittleEndian(progress, 1); + BinaryPrimitives.WriteUInt64LittleEndian(progress[sizeof(ulong)..], 1); return ctx.Memory.TryWrite(outProgress, progress) ? (int)OrbisGen2Result.ORBIS_GEN2_OK : (int)OrbisGen2Result.ORBIS_GEN2_ERROR_MEMORY_FAULT; @@ -646,6 +646,32 @@ public static class PlayGoExports : (int)OrbisGen2Result.ORBIS_GEN2_OK; } + [SysAbiExport( + Nid = "RHXYK0iDICM", + ExportName = "scePlayGoRequestNextChunk", + Target = Generation.Gen4 | Generation.Gen5, + LibraryName = "libScePlayGo")] + public static int PlayGoRequestNextChunk(CpuContext ctx) + { + var handle = unchecked((uint)ctx[CpuRegister.Rdi]); + var outChunkId = ctx[CpuRegister.Rsi]; + + var validation = ValidateHandle(handle); + if (validation != 0) + { + return validation; + } + + if (outChunkId == 0) + { + return OrbisPlayGoErrorBadPointer; + } + + return ctx.TryWriteUInt16(outChunkId, 0) + ? OrbisKernelErrorNoEntry + : (int)OrbisGen2Result.ORBIS_GEN2_ERROR_MEMORY_FAULT; + } + [SysAbiExport( Nid = "4AAcTU9R3XM", ExportName = "scePlayGoSetInstallSpeed", @@ -763,7 +789,7 @@ public static class PlayGoExports { lock (_stateGate) { - return _metadata.ChunkIdKnowledge == PlayGoChunkIdKnowledge.Unknown || + return _metadata.ChunkIdKnowledge != PlayGoChunkIdKnowledge.Authoritative || Array.BinarySearch(_metadata.ChunkIds, chunkId) >= 0; } } @@ -778,48 +804,100 @@ public static class PlayGoExports return CreateBaseChunkMetadata(PlayGoChunkIdKnowledge.Authoritative); } + var playGoPgm = Path.Combine(app0Root, "cache_ps5", "playgo.pgm"); var playGoDat = Path.Combine(app0Root, "sce_sys", "playgo-chunk.dat"); var scenarioJson = Path.Combine(app0Root, "sce_sys", "playgo-scenario.json"); var chunkDefsXml = Path.Combine(app0Root, "playgo-chunkdefs.xml"); - var hasMetadata = File.Exists(playGoDat) || File.Exists(scenarioJson) || File.Exists(chunkDefsXml); - if (!hasMetadata) + var pgmChunkIds = LoadPlayGoPgmChunkIds(playGoPgm); + if (pgmChunkIds.Length > 0) { - // No PlayGo sidecar: derive the installed chunk set from the pak files - // actually present on disk. A locally dumped title has all of its data - // installed, and a package that splits content across chunks names them - // pakchunk-.pak, so those N are exactly the chunks that - // exist. Reporting only chunk 0 told such a title its remaining content - // was missing: The Invincible (PPSA06426) ships pakchunk0..8 and spun - // forever re-querying scePlayGoGetLocus for a chunk that never became - // available. Available must stay true or scePlayGoOpen fails with - // NotSupportPlayGo (fatal PS5-component init failure for UE titles). - // Ids outside the discovered set still return BAD_CHUNK_ID, so - // title-side chunk enumeration still terminates. - var installedChunkIds = DiscoverInstalledChunkIds(app0Root); - TracePlayGo($"metadata_missing; fully-installed chunks=[{string.Join(',', installedChunkIds)}]"); - return new PlayGoMetadata( - true, - installedChunkIds, - PlayGoChunkIdKnowledge.Authoritative); + TracePlayGo($"metadata_pgm chunks={pgmChunkIds.Length}"); + return new PlayGoMetadata(true, pgmChunkIds, PlayGoChunkIdKnowledge.Assumed); } var chunkIds = LoadChunkIds(chunkDefsXml); - if (chunkIds.Length == 0) + if (chunkIds.Length > 0) { - TracePlayGo("metadata_has_no_chunk_definitions; installed chunk ids unknown"); + TracePlayGo($"metadata_chunkdefs chunks=[{string.Join(',', chunkIds)}]"); + return new PlayGoMetadata(true, chunkIds, PlayGoChunkIdKnowledge.Authoritative); + } + + if (File.Exists(playGoDat) || File.Exists(scenarioJson) || File.Exists(chunkDefsXml)) + { + // These sidecars are used by PS4 and by scenarios whose chunk map + // is not understood here. Their presence proves PlayGo is in use, + // but does not prove that the derived chunk list is complete. Keep + // the pre-pgm permissive behaviour so titles such as The Invincible + // do not receive BAD_CHUNK_ID for content they own. + TracePlayGo("metadata_present_but_unparsed; chunk_ids_unknown"); return new PlayGoMetadata( true, Array.Empty(), PlayGoChunkIdKnowledge.Unknown); } + // Derive the installed chunk set from pak files when no usable chunk map + // was available. A locally dumped title has all of its data installed, and + // chunk 0 is always retained as the final single-chunk fallback. + var installedChunkIds = DiscoverInstalledChunkIds(app0Root); + TracePlayGo($"metadata_fallback; fully-installed chunks=[{string.Join(',', installedChunkIds)}]"); return new PlayGoMetadata( true, - chunkIds, + installedChunkIds, PlayGoChunkIdKnowledge.Authoritative); } + private static ushort[] LoadPlayGoPgmChunkIds(string playGoPgm) + { + Span header = stackalloc byte[0x14]; + try + { + using var stream = File.OpenRead(playGoPgm); + if (stream.Length < header.Length) + { + return Array.Empty(); + } + + stream.ReadExactly(header); + } + catch (FileNotFoundException) + { + return Array.Empty(); + } + catch (DirectoryNotFoundException) + { + return Array.Empty(); + } + catch (IOException) + { + return Array.Empty(); + } + catch (UnauthorizedAccessException) + { + return Array.Empty(); + } + + if (!header[..4].SequenceEqual("DMGP"u8)) + { + return Array.Empty(); + } + + var count = BinaryPrimitives.ReadUInt32LittleEndian(header[0x10..]); + if (count == 0 || count > (uint)ushort.MaxValue + 1u) + { + return Array.Empty(); + } + + var chunkIds = new ushort[checked((int)count)]; + for (var index = 0; index < chunkIds.Length; index++) + { + chunkIds[index] = (ushort)index; + } + + return chunkIds; + } + private static PlayGoMetadata CreateBaseChunkMetadata(PlayGoChunkIdKnowledge knowledge) => new( true, @@ -952,6 +1030,9 @@ public static class PlayGoExports { Unknown, Authoritative, + // The list is derived rather than read (playgo.pgm gives only a count), so it is + // reported but an ID outside it is not rejected. + Assumed, } private sealed record PlayGoMetadata( diff --git a/src/SharpEmu.Libs/SystemService/SystemServiceExports.cs b/src/SharpEmu.Libs/SystemService/SystemServiceExports.cs index c56e059..5f69833 100644 --- a/src/SharpEmu.Libs/SystemService/SystemServiceExports.cs +++ b/src/SharpEmu.Libs/SystemService/SystemServiceExports.cs @@ -11,6 +11,7 @@ namespace SharpEmu.Libs.SystemService; public static class SystemServiceExports { private const int OrbisSystemServiceErrorParameter = unchecked((int)0x80A10003); + private const int OrbisSystemServiceErrorNoEvent = unchecked((int)0x80A10004); private const int SystemServiceStatusSize = 0x0C; private const int DisplaySafeAreaInfoSize = sizeof(float) + 128; private const int HdrToneMapLuminanceSize = sizeof(float) * 3; @@ -21,6 +22,21 @@ public static class SystemServiceExports private static int _noticeScreenSkipFlag; private static int _systemLanguage = 1; + [SysAbiExport( + Nid = "656LMQSrg6U", + ExportName = "sceSystemServiceReceiveEvent", + Target = Generation.Gen4 | Generation.Gen5, + LibraryName = "libSceSystemService")] + public static int SystemServiceReceiveEvent(CpuContext ctx) + { + if (ctx[CpuRegister.Rdi] == 0) + return ctx.SetReturn(OrbisSystemServiceErrorParameter); + + // No host system-service events are generated yet. An empty poll must + // leave the caller's event record untouched and report NO_EVENT. + return ctx.SetReturn(OrbisSystemServiceErrorNoEvent); + } + public static void ConfigureApplicationInfo(string? titleId, int systemLanguage = 1) { _mainAppTitleId = string.IsNullOrWhiteSpace(titleId) ? null : titleId.Trim(); diff --git a/src/SharpEmu.Libs/VideoOut/HostVideoOptions.cs b/src/SharpEmu.Libs/VideoOut/HostVideoOptions.cs index b0a8f9c..b0bca72 100644 --- a/src/SharpEmu.Libs/VideoOut/HostVideoOptions.cs +++ b/src/SharpEmu.Libs/VideoOut/HostVideoOptions.cs @@ -21,6 +21,13 @@ public enum HostScalingMode Integer, } +// The display mode exposed to the guest, independent of host window and guest buffer sizes. +public enum GuestDisplayResolution +{ + Hd = 1, + UltraHd = 2, +} + public enum HostHdrMode { Auto, @@ -55,6 +62,12 @@ public sealed record HostVideoOptions public int Height { get; init; } = 1080; + public GuestDisplayResolution GuestResolution { get; init; } = GuestDisplayResolution.Hd; + + // Host resolution multiplier of the guest render targets the scaling rules accept. + // One renders every target at the size the game asked for. + public float RenderScale { get; init; } = 1.0f; + public int DisplayIndex { get; init; } public int RefreshRate { get; init; } @@ -78,10 +91,20 @@ public sealed record HostVideoOptions internal bool CanUseHdr(bool displayHdrEnabled, bool surfaceSupportsHdr, bool autoAllowed) => surfaceSupportsHdr && (HdrMode == HostHdrMode.On || HdrMode == HostHdrMode.Auto && autoAllowed && displayHdrEnabled); + public const float MinimumRenderScale = 0.25f; + public const float MaximumRenderScale = 2.0f; + + // Rounds to a hundredth so the GUI, the command line and the environment agree bit for bit; + // the renderer compares scales for equality when it decides whether two images alias. + public static float NormalizeRenderScale(float scale) => + !float.IsFinite(scale) ? 1.0f : MathF.Round(Math.Clamp(scale, MinimumRenderScale, MaximumRenderScale), 2); + public HostVideoOptions Normalize() => this with { Width = Math.Clamp(Width, 640, 16384), Height = Math.Clamp(Height, 360, 16384), + GuestResolution = Enum.IsDefined(GuestResolution) ? GuestResolution : GuestDisplayResolution.Hd, + RenderScale = NormalizeRenderScale(RenderScale), DisplayIndex = Math.Max(0, DisplayIndex), RefreshRate = Math.Clamp(RefreshRate, 0, 1000), HdrMode = Enum.IsDefined(HdrMode) ? HdrMode : HostHdrMode.Auto, @@ -105,6 +128,7 @@ public static class HostVideoHost { var normalized = options.Normalize(); Volatile.Write(ref _currentOptions, normalized); + Gpu.Images.RenderScalePolicy.Configure(normalized.RenderScale); var configured = VulkanVideoPresenter.TryConfigureVideo(normalized) & MetalVideoPresenter.TryConfigureVideo(normalized); if (configured) diff --git a/src/SharpEmu.Libs/VideoOut/RenderDocCapture.cs b/src/SharpEmu.Libs/VideoOut/RenderDocCapture.cs index b9dcc7f..1362885 100644 --- a/src/SharpEmu.Libs/VideoOut/RenderDocCapture.cs +++ b/src/SharpEmu.Libs/VideoOut/RenderDocCapture.cs @@ -148,6 +148,8 @@ public static unsafe class RenderDocCapture } } + private static readonly string TriggerFile = Path.Combine(AppContext.BaseDirectory, "renderdoc.capture"); + public static void RequestCapture() { if (_api is null) @@ -182,6 +184,14 @@ public static unsafe class RenderDocCapture return; } + // Unattended runs request a capture by creating this file next to the executable. + if (state == StateIdle && File.Exists(TriggerFile)) + { + try { File.Delete(TriggerFile); } catch (IOException) { } + RequestCapture(); + state = Volatile.Read(ref _state); + } + if (state != StateRequested) { return; diff --git a/src/SharpEmu.Libs/VideoOut/RenderPhaseProfile.cs b/src/SharpEmu.Libs/VideoOut/RenderPhaseProfile.cs index 8821b5d..8041cb1 100644 --- a/src/SharpEmu.Libs/VideoOut/RenderPhaseProfile.cs +++ b/src/SharpEmu.Libs/VideoOut/RenderPhaseProfile.cs @@ -467,8 +467,13 @@ internal static class RenderPhaseProfile ReportImageUploads(); BufferUploadProfile.Report(); + if (SharpEmu.Libs.Gpu.Rendering.TessellationProfile.TakeReport() is { } tessellationReport) + { + Console.Error.WriteLine(tessellationReport); + } Console.Error.WriteLine(SharpEmu.ShaderCompiler.Resources.ResourceMaterializationCache.TakeReport()); Console.Error.WriteLine(SharpEmu.Libs.Gpu.Buffers.GuestBufferCache.TakeAsyncReadbackReport()); + Console.Error.WriteLine(SharpEmu.Libs.Gpu.Buffers.GuestBufferCache.TakeDeviceAddressSyncReport()); Console.Error.WriteLine(SharpEmu.Libs.Gpu.Images.GuestImageCache.TakeLookupReport()); SharpEmu.ShaderCompiler.Resources.ResourceMaterializationProfile.WriteReport(); SharpEmu.Libs.Diagnostics.AgcRegisterPacketProfile.WriteReport(); diff --git a/src/SharpEmu.Libs/VideoOut/SdlHostWindow.cs b/src/SharpEmu.Libs/VideoOut/SdlHostWindow.cs index 8f6ff76..7888e57 100644 --- a/src/SharpEmu.Libs/VideoOut/SdlHostWindow.cs +++ b/src/SharpEmu.Libs/VideoOut/SdlHostWindow.cs @@ -35,6 +35,7 @@ internal sealed unsafe class SdlHostWindow : IDisposable, IHostGamepadOutput private SDL_Window* _window; private nint _metalView; private SDL_Gamepad* _gamepad; + private readonly Pad.TouchContactIds _touchIds = new(); private HostGamepadType _gamepadType; private byte _leftTriggerRumble; private byte _rightTriggerRumble; @@ -808,6 +809,12 @@ internal sealed unsafe class SdlHostWindow : IDisposable, IHostGamepadOutput } private HostTouchPoint ReadTouchPoint(int finger) + { + var point = ReadRawTouchPoint(finger); + return point with { Id = _touchIds.Track(finger, point.Active) }; + } + + private HostTouchPoint ReadRawTouchPoint(int finger) { if (SDL_GetNumGamepadTouchpadFingers(_gamepad, 0) <= finger) { diff --git a/src/SharpEmu.Libs/VideoOut/VideoOutExports.FlipRequests.cs b/src/SharpEmu.Libs/VideoOut/VideoOutExports.FlipRequests.cs index 1181350..375026b 100644 --- a/src/SharpEmu.Libs/VideoOut/VideoOutExports.FlipRequests.cs +++ b/src/SharpEmu.Libs/VideoOut/VideoOutExports.FlipRequests.cs @@ -37,6 +37,8 @@ public static partial class VideoOutExports public int FlipMode; public long FlipArg; public bool GpuQueued; + // In command-stream order: a GPU flip from its packet, a CPU flip once its preparation runs on the graphics queue. + public bool Ordered; public long? ReadyTimestamp; public ulong EventHint; public FlipEventRegistration[]? FlipEvents; @@ -128,6 +130,7 @@ public static partial class VideoOutExports FlipMode = flipMode, FlipArg = flipArg, GpuQueued = gpuQueued, + Ordered = gpuQueued, EventHint = SceVideoOutInternalEventFlip | ((unchecked((ulong)flipArg) & 0x0000_FFFF_FFFF_FFFFUL) << 16), FlipEventCount = port.FlipEvents.Count, @@ -296,14 +299,30 @@ public static partial class VideoOutExports } } + // The graphics queue reached the preparation of a CPU flip, so later waits on its buffer are ordered after it. + internal static void MarkFlipOrdered(ulong requestId) + { + lock (_stateGate) + { + if (_flipRequests.TryGetValue(requestId, out var request)) + { + request.Ordered = true; + } + } + } + // True when no flip still waits for the presenter on the buffer; a discarded or cancelled flip no longer does. + // A CPU flip counts only once the graphics queue has reached it: sceVideoOutSubmitFlip reserves the request + // (and may sleep for pacing) before queueing its preparation, so a command buffer submitted earlier must not + // wait for it; the preparation sits behind that command buffer and the wait would never end. internal static bool IsFlipDone(int handle, int bufferIndex) { lock (_stateGate) { foreach (var request in _flipRequests.Values) { - if (request.Handle == handle && request.BufferIndex == bufferIndex && request.Outcome == FlipOutcome.Pending) + if (request.Handle == handle && request.BufferIndex == bufferIndex && request.Ordered && + request.Outcome == FlipOutcome.Pending) { return false; } diff --git a/src/SharpEmu.Libs/VideoOut/VideoOutExports.cs b/src/SharpEmu.Libs/VideoOut/VideoOutExports.cs index 2688f09..25eec50 100644 --- a/src/SharpEmu.Libs/VideoOut/VideoOutExports.cs +++ b/src/SharpEmu.Libs/VideoOut/VideoOutExports.cs @@ -519,18 +519,19 @@ public static partial class VideoOutExports } Span status = stackalloc byte[VideoOutOutputStatusSize]; - WriteOutputStatus(status, port.OutputWidth, port.OutputHeight, port.RefreshRate, + WriteOutputStatus(status, HostVideoHost.CurrentOptions.GuestResolution, port.RefreshRate, HostVideoHost.IsHdrOutputSupported); return ctx.Memory.TryWrite(statusAddress, status) ? (int)OrbisGen2Result.ORBIS_GEN2_OK : (int)OrbisGen2Result.ORBIS_GEN2_ERROR_MEMORY_FAULT; } - internal static void WriteOutputStatus(Span status, uint width, uint height, uint refreshRate, + internal static void WriteOutputStatus(Span status, GuestDisplayResolution resolution, uint refreshRate, bool hdrOutputSupported) { status.Clear(); - var resolutionClass = width >= 3840 || height >= 2160 ? 2 : 1; + // Buffer registration describes a source image, not the connected display. + var resolutionClass = (int)resolution; BinaryPrimitives.WriteInt32LittleEndian(status[0x00..0x04], resolutionClass); BinaryPrimitives.WriteInt32LittleEndian(status[0x04..0x08], hdrOutputSupported ? 2 : 1); // The status uses a refresh-rate code, not the frequency used for pacing. diff --git a/src/SharpEmu.Libs/VideoOut/VulkanFloat16Support.cs b/src/SharpEmu.Libs/VideoOut/VulkanFloat16Support.cs new file mode 100644 index 0000000..e2b3c0d --- /dev/null +++ b/src/SharpEmu.Libs/VideoOut/VulkanFloat16Support.cs @@ -0,0 +1,42 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using Silk.NET.Vulkan; + +namespace SharpEmu.Libs.VideoOut; + +// Whether a device can run f16<->f32 conversions natively with the hardware's results: +// shaderFloat16, plus round-to-nearest-even and denormal preservation for 16-bit floats +// (float controls), including signed zero, infinity and NaN preservation. Shaders can +// then use both native conversions and fused half arithmetic with explicit float controls. +public static unsafe class VulkanFloat16Support +{ + public static bool SupportsExactConversions(Vk vk, PhysicalDevice physical) + { + var float16Features = new PhysicalDeviceShaderFloat16Int8Features + { + SType = StructureType.PhysicalDeviceShaderFloat16Int8Features, + }; + var features = new PhysicalDeviceFeatures2 + { + SType = StructureType.PhysicalDeviceFeatures2, + PNext = &float16Features, + }; + vk.GetPhysicalDeviceFeatures2(physical, &features); + + var floatControls = new PhysicalDeviceFloatControlsProperties + { + SType = StructureType.PhysicalDeviceFloatControlsProperties, + }; + var properties = new PhysicalDeviceProperties2 + { + SType = StructureType.PhysicalDeviceProperties2, + PNext = &floatControls, + }; + vk.GetPhysicalDeviceProperties2(physical, &properties); + return float16Features.ShaderFloat16 && + floatControls.ShaderRoundingModeRtefloat16 && + floatControls.ShaderDenormPreserveFloat16 && + floatControls.ShaderSignedZeroInfNanPreserveFloat16; + } +} diff --git a/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.CachedImages.cs b/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.CachedImages.cs index 8d57211..415a71a 100644 --- a/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.CachedImages.cs +++ b/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.CachedImages.cs @@ -115,6 +115,7 @@ internal static unsafe partial class VulkanVideoPresenter public Sampler Sampler; public GuestSampler SamplerState; public bool IsStorage; + public bool IsResident; public bool IsHostMovie; public int HostMoviePlane = -1; public long HostMovieFrameSerial; @@ -186,6 +187,10 @@ internal static unsafe partial class VulkanVideoPresenter TrackImageBinding(imageIdentifier); } + public void DemoteRenderScale(ResourceSlotIdentifier imageIdentifier) => _imageCache.DemoteRenderScale(imageIdentifier); + + public float GetRenderScale(ResourceSlotIdentifier imageIdentifier) => _imageCache.GetImage(imageIdentifier).RenderScale; + private void ResetImageBindings() { foreach (var imageIdentifier in _trackedImageBindings) @@ -380,34 +385,18 @@ internal static unsafe partial class VulkanVideoPresenter } } + // The color a DCC clear code decompresses to: the register clear (0x20) needs the + // target's clear words, the fixed codes need a format that encodes them. private static bool TryDecodeDccClear(byte code, bool registerClearSupported, bool fixedClearSupported, ClearColorValue registerClear, out ClearColorValue clearValue) { - clearValue = default; - switch (code) + if (code == 0x20) { - case 0x00: - return true; - case 0x20: - clearValue = registerClear; - return registerClearSupported; - case 0x40: - clearValue.Float32_3 = 1f; - return fixedClearSupported; - case 0x80: - clearValue.Float32_0 = 1f; - clearValue.Float32_1 = 1f; - clearValue.Float32_2 = 1f; - return fixedClearSupported; - case 0xc0: - clearValue.Float32_0 = 1f; - clearValue.Float32_1 = 1f; - clearValue.Float32_2 = 1f; - clearValue.Float32_3 = 1f; - return fixedClearSupported; - default: - return false; + clearValue = registerClear; + return registerClearSupported; } + + return ImageRequestBuilders.TryFixedDccClearValue(code, out clearValue) && (code == 0x00 || fixedClearSupported); } private DepthAttachment? DiscoverDepthTarget(GuestDepthTarget target) @@ -579,6 +568,7 @@ internal static unsafe partial class VulkanVideoPresenter Request = request, MipLevel = texture.MipLevel, IsStorage = texture.IsStorage, + IsResident = true, SamplerState = texture.Sampler, DestinationSelect = texture.DstSelect, Width = texture.Width, @@ -650,12 +640,19 @@ internal static unsafe partial class VulkanVideoPresenter private CachedImage AcquireStencilStorage(TextureResource binding, CachedImage attachment, bool writeBack = true) { + // The view can be smaller than the attachment (a dynamic-resolution pass writes only + // the region it renders); the storage image then has the view's size. var description = binding.Request.Description; if (description.Data.Address != attachment.Description.Stencil.Address || description.Data.Size > attachment.Description.Stencil.Size || - description.Extent.Width != attachment.Backing.Extent.Width || description.Extent.Height != attachment.Backing.Extent.Height || + description.Extent.Width > attachment.Backing.Extent.Width || description.Extent.Height > attachment.Backing.Extent.Height || description.Resources.Levels != 1 || description.Resources.Layers != attachment.Backing.Layers) { - throw SubmissionScheduler.Fatal("The stencil storage request does not cover its attachment's stencil layout."); + throw SubmissionScheduler.Fatal( + $"The stencil storage request does not fit its attachment's stencil layout: " + + $"request=0x{description.Data.Address:X}+0x{description.Data.Size:X} {description.Extent.Width}x{description.Extent.Height} " + + $"levels={description.Resources.Levels} layers={description.Resources.Layers} " + + $"stencil=0x{attachment.Description.Stencil.Address:X}+0x{attachment.Description.Stencil.Size:X} " + + $"attachment={attachment.Backing.Extent.Width}x{attachment.Backing.Extent.Height} layers={attachment.Backing.Layers}."); } if (writeBack && _hasBoundDepth && _boundDepth.Image == binding.ImageIdentifier && @@ -668,6 +665,13 @@ internal static unsafe partial class VulkanVideoPresenter var stencilImages = preparation.StencilStorageImages ??= new(); if (stencilImages.TryGetValue(attachment, out var storage)) { + if (storage.Backing.Extent.Width != description.Extent.Width || storage.Backing.Extent.Height != description.Extent.Height) + { + throw SubmissionScheduler.Fatal( + $"Two stencil storage views of one attachment in a draw differ in size: " + + $"{storage.Backing.Extent.Width}x{storage.Backing.Extent.Height} and {description.Extent.Width}x{description.Extent.Height}."); + } + if (writeBack) { preparation.StencilStorageWriteBackImages ??= new(); @@ -689,7 +693,7 @@ internal static unsafe partial class VulkanVideoPresenter _imageCache.MarkGpuWritten(binding.ImageIdentifier); } - storage = attachment.CreateStencilStorageImage(); + storage = _imageCache.AcquireStencilStorageImage(binding.ImageIdentifier, description.Extent.Width, description.Extent.Height); stencilImages.Add(attachment, storage); storage.Binding.ShaderWrite = writeBack; if (writeBack) @@ -1162,6 +1166,12 @@ internal static unsafe partial class VulkanVideoPresenter } } + // Retired flip snapshots kept for the next flips. Every flip copies the display surface + // into a snapshot of the same size and format; creating and freeing a 4K image per flip + // was hundreds of megabytes of driver allocations a second. + private const int MaxRecycledFlipSnapshots = 4; + private readonly List _recycledFlipSnapshots = []; + private void DestroyGuestImage(GuestImageResource resource) { if (TryPoolFlipSnapshot(resource)) diff --git a/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.CommandStream.cs b/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.CommandStream.cs index 36e741d..b7a8209 100644 --- a/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.CommandStream.cs +++ b/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.CommandStream.cs @@ -290,6 +290,7 @@ internal static unsafe partial class VulkanVideoPresenter public bool TryReadGuest(ulong address, Span destination) { + if (_bufferCache.TryReadCommandBacking(address, destination)) return true; using (RenderPhaseProfile.MeasureDetail(RenderPhaseProfile.Phase.CommandMemorySync)) { if (!_bufferCache.TrySynchronizeCpuRead(address, (ulong)destination.Length, @@ -315,9 +316,14 @@ internal static unsafe partial class VulkanVideoPresenter _activeGuestQueue = new VulkanGuestQueueIdentity(_commandQueueNames[queueId], submissionId); BindSubmissionContext(_activeGuestQueue); _ = CurrentRecordingBuffer(); + _bufferCache.NoteMemoryVisibilityPoint(); _translation.BeginSubmission(queueId, submissionId, geometrySnapshots, _commandStream.GetInterpreter(queueId)); } + public void NoteMemoryVisibilityPoint() => _bufferCache.NoteMemoryVisibilityPoint(); + + public void NoteCommandProcessorWrite(ulong address, ulong size) => _bufferCache.NoteCommandProcessorWrite(address, size); + public void Flush() { using var flushScope = RenderPhaseProfile.MeasureDetail(RenderPhaseProfile.Phase.Flush); @@ -357,6 +363,20 @@ internal static unsafe partial class VulkanVideoPresenter RecordGlobalBarrier(BeginBatchedGuestCommands()); } + // SHARPEMU_DIAG_BARRIER_EVERY_CALL=1 (diagnostic): a full memory barrier before every draw and + // dispatch, so no shader can overlap or read past an earlier one's writes. Very slow. + private static readonly bool BarrierEveryCall = + Environment.GetEnvironmentVariable("SHARPEMU_DIAG_BARRIER_EVERY_CALL") == "1"; + + private void DiagnosticBarrier() + { + if (BarrierEveryCall) + { + EndRendering(); + RecordGlobalBarrier(BeginBatchedGuestCommands()); + } + } + private void RecordGlobalBarrier(CommandBuffer commandBuffer) { var barrier = new MemoryBarrier2 @@ -474,8 +494,11 @@ internal static unsafe partial class VulkanVideoPresenter public bool IsFlipDone(int handle, int index) => VideoOutExports.IsFlipDone(handle, index); - public void PrepareCpuFlip(int handle, int index, ulong requestId) => + public void PrepareCpuFlip(int handle, int index, ulong requestId) + { + VideoOutExports.MarkFlipOrdered(requestId); CaptureFlip(handle, index, requestId, flipMode: 0, flipArg: 0); + } public void DrawIndexed(ulong submitId, in DrawIndexedArguments arguments) { @@ -483,6 +506,7 @@ internal static unsafe partial class VulkanVideoPresenter var started = System.Diagnostics.Stopwatch.GetTimestamp(); try { + DiagnosticBarrier(); _translation.DrawIndexed(submitId, in arguments); } finally @@ -497,6 +521,7 @@ internal static unsafe partial class VulkanVideoPresenter var started = System.Diagnostics.Stopwatch.GetTimestamp(); try { + DiagnosticBarrier(); _translation.DrawAuto(submitId, in arguments); } finally @@ -513,6 +538,9 @@ internal static unsafe partial class VulkanVideoPresenter // SHARPEMU_CPU_INDIRECT_DRAW=1 reads the arguments back on the CPU as before. public bool ResolvesIndirectDrawOnGpu => !_cpuIndirectDraw; + // RenderExecutor records non-indexed indirect draws with vkCmdDrawIndirect. + public bool ResolvesNonIndexedIndirectDrawOnGpu => !_cpuIndirectDraw; + private static readonly bool _cpuIndirectDraw = string.Equals( Environment.GetEnvironmentVariable("SHARPEMU_CPU_INDIRECT_DRAW"), "1", StringComparison.Ordinal); @@ -525,6 +553,7 @@ internal static unsafe partial class VulkanVideoPresenter var started = System.Diagnostics.Stopwatch.GetTimestamp(); try { + DiagnosticBarrier(); _translation.Dispatch(submitId, groupsX, groupsY, groupsZ, dispatchInitiator, indirectArgumentsAddress); } finally @@ -753,6 +782,22 @@ internal static unsafe partial class VulkanVideoPresenter _commandStream.RetryBlocked(); } + while (_pendingVideoPresentations.Count > 0 && + _pendingVideoPresentations.Peek().Sequence <= _presentedSequence) + { + _pendingVideoPresentations.Dequeue(); + } + + // Both queues share a sequence. A later guest flip must not retire + // an earlier decoded frame, even while that flip waits for its GPU tick. + if (_pendingVideoPresentations.Count > 0 && + (_pendingGuestImagePresentations.Count == 0 || + _pendingVideoPresentations.Peek().Sequence < _pendingGuestImagePresentations.Peek().Sequence)) + { + presentation = _pendingVideoPresentations.Dequeue(); + return true; + } + if (_pendingGuestImagePresentations.Count > 0) { var pending = _pendingGuestImagePresentations.Peek(); @@ -767,12 +812,6 @@ internal static unsafe partial class VulkanVideoPresenter return false; } - while (_pendingVideoPresentations.Count > 0 && - _pendingVideoPresentations.Peek().Sequence <= _presentedSequence) - { - _pendingVideoPresentations.Dequeue(); - } - if (_pendingVideoPresentations.Count > 0) { presentation = _pendingVideoPresentations.Dequeue(); @@ -807,6 +846,13 @@ internal static unsafe partial class VulkanVideoPresenter // True when a guest frame can be shown now; the wait loop wakes for it. private bool HasReadyPresentationLocked() { + if (_pendingVideoPresentations.Count > 0 && + (_pendingGuestImagePresentations.Count == 0 || + _pendingVideoPresentations.Peek().Sequence < _pendingGuestImagePresentations.Peek().Sequence)) + { + return true; + } + if (_pendingGuestImagePresentations.Count > 0) { var pending = _pendingGuestImagePresentations.Peek(); diff --git a/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.Descriptors.cs b/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.Descriptors.cs index cdc3b6d..45b71c4 100644 --- a/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.Descriptors.cs +++ b/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.Descriptors.cs @@ -50,6 +50,8 @@ internal static unsafe partial class VulkanVideoPresenter public BufferView GlobalDataShare; public BufferView FlattenedTable; public BufferView ShaderData; + public BufferView RuntimeTable; + public BufferView RuntimeMisses; } private sealed class PreparedStageBindings(ShaderStageResources stage, ShaderProgramInfo program) : IPreparedBindings @@ -68,7 +70,13 @@ internal static unsafe partial class VulkanVideoPresenter public uint[] ShaderData { get; set; } = []; - public TextureResource[] Textures => Descriptors.Images; + // The registered runtime-descriptor images the draw samples through the heap; they + // take the same transitions as the bound images but no binding of their own. + public List RuntimeImages { get; } = []; + + public List RuntimeEntries { get; } = []; + + public TextureResource[] Textures => RuntimeImages.Count == 0 ? Descriptors.Images : [.. Descriptors.Images, .. RuntimeImages]; } private static ShaderStage StageOf(ShaderProgramInfo program) => program.Stage switch @@ -76,6 +84,7 @@ internal static unsafe partial class VulkanVideoPresenter ShaderStageKind.Vertex => ShaderStage.Vertex, ShaderStageKind.Pixel => ShaderStage.Pixel, ShaderStageKind.Compute => ShaderStage.Compute, + ShaderStageKind.TessellationEvaluation => ShaderStage.TessellationEvaluation, _ => throw SubmissionScheduler.Fatal($"The stage kind is unknown: stage={program.Stage} hash=0x{program.Hash:X16}."), }; @@ -110,6 +119,101 @@ internal static unsafe partial class VulkanVideoPresenter DepthCompare: image.DepthCompare, Atomic: image.Atomic); + private static bool[] FindResidentBindlessImages(ShaderResourceInfo info, uint[] flattenedTable) + { + var resident = new bool[info.Images.Count]; + for (var index = 0; index < info.Images.Count; index++) + { + if (info.Images[index].IndirectRoot == DescriptorConstants.NoIndex) + { + resident[index] = true; + } + } + + for (var rootIndex = 0; rootIndex < info.Images.Count; rootIndex++) + { + var root = info.Images[rootIndex]; + if (root.IndirectRoot != (uint)rootIndex || root.IndirectSearchIterations == 0) + { + continue; + } + + var candidates = root.IndirectResources; + var offset = root.IndirectMappingOffset; + if (candidates.Count == 0 || offset >= flattenedTable.Length) + { + // An incomplete mapping is not a valid basis for dropping a + // descriptor: preserve the old eager behavior for this table. + foreach (var candidate in candidates) + { + if (candidate < resident.Length) + { + resident[(int)candidate] = true; + } + } + + if (candidates.Count == 0) + { + resident[rootIndex] = true; + } + + continue; + } + + var count = flattenedTable[offset]; + if (count == 0 || (ulong)offset + 1 + (ulong)count * 2 > (ulong)flattenedTable.Length) + { + foreach (var candidate in candidates) + { + if (candidate < resident.Length) + { + resident[(int)candidate] = true; + } + } + + continue; + } + + for (var entry = 0u; entry < count; entry++) + { + var candidate = flattenedTable[offset + 2 + entry * 2]; + if (candidate < candidates.Count) + { + var resource = candidates[(int)candidate]; + if (resource < resident.Length) + { + resident[(int)resource] = true; + } + } + } + } + + return resident; + } + + private TextureResource ResolveNonResidentImage(ImageResource image, uint[] words, ShaderProgramInfo program, int index) + { + var descriptor = new TextureDescriptorWords(words); + var resolution = ImageRequestBuilders.NullTextureResolution(ShapeOf(image)); + _ = BeginBatchedGuestCommands(); + var request = resolution.Request; + var imageIdentifier = _imageCache.FindImage(ref request); + resolution = resolution with { Request = request }; + imageIdentifier = ImageRequestBuilders.ValidateTextureOwner(_imageCache, imageIdentifier, resolution); + BindImage(imageIdentifier, image.ResourceClass == ShaderCompiler.Resources.ImageResourceClass.Storage); + return new TextureResource + { + Address = descriptor.BaseAddress, + ImageIdentifier = imageIdentifier, + Request = request, + IsStorage = image.ResourceClass == ShaderCompiler.Resources.ImageResourceClass.Storage, + IsResident = true, + Width = descriptor.Width, + Height = descriptor.Height, + DestinationSelect = resolution.Swizzle, + }; + } + private static readonly System.Collections.Concurrent.ConcurrentDictionary _tracedTextureBindings = new(); // Render-state discovery for one shader image; the view is acquired later with the draw. @@ -124,6 +228,7 @@ internal static unsafe partial class VulkanVideoPresenter var resolution = ImageRequestBuilders.Texture(words, ShapeOf(image)); _ = BeginBatchedGuestCommands(); var request = resolution.Request; + request.ShaderWrite = storage && (image.Written || image.Atomic); var imageIdentifier = _imageCache.FindImage(ref request, resolution.ExactFormat); resolution = resolution with { Request = request }; imageIdentifier = ImageRequestBuilders.ValidateTextureOwner(_imageCache, imageIdentifier, resolution); @@ -157,6 +262,7 @@ internal static unsafe partial class VulkanVideoPresenter ImageIdentifier = imageIdentifier, Request = request, IsStorage = storage, + IsResident = true, DestinationSelect = words[3] & 0xFFFu, Width = descriptor.Width, Height = descriptor.Height, @@ -171,7 +277,7 @@ internal static unsafe partial class VulkanVideoPresenter var paired = false; foreach (var pair in info.SampledPairs) { - if (pair.Sampler != sampler || pair.Image >= images.Length || images[pair.Image].IsHostMovie) + if (pair.Sampler != sampler || pair.Image >= images.Length || images[pair.Image].IsHostMovie || !images[pair.Image].IsResident) { continue; } @@ -266,13 +372,21 @@ internal static unsafe partial class VulkanVideoPresenter descriptors.Images = Scratch.Images.Rent(info.Images.Count); var movieCandidates = _hostMovieFramePixels is null ? null : MovieCandidates(resources, snapshot); var hostMovie = movieCandidates is null ? HostMovieTextureBindings.None : FindHostMovieTextureBindings(movieCandidates); + var residentImages = layout.UsesBindlessImages ? FindResidentBindlessImages(info, snapshot.FlattenedResourceTable) : null; for (var index = 0; index < info.Images.Count; index++) { descriptors.Images[index] = index == hostMovie.Luma ? CreateHostMovieTextureResource(movieCandidates![index], plane: 0) : index == hostMovie.Chroma ? CreateHostMovieTextureResource(movieCandidates![index], plane: 1) - : ResolveImageBinding(info.Images[index], snapshot.Images[index], program, index); + : residentImages is { } && !residentImages[index] + ? ResolveNonResidentImage(info.Images[index], snapshot.Images[index], program, index) + : ResolveImageBinding(info.Images[index], snapshot.Images[index], program, index); + } + + if (layout.Find(DescriptorBindingKind.RuntimeDescriptorTable) is not null) + { + PrepareRuntimeDescriptors(prepared); } descriptors.Samplers = Scratch.Samplers.Rent(info.Samplers.Count); @@ -303,6 +417,23 @@ internal static unsafe partial class VulkanVideoPresenter } stage.WriteDispatchThreadLimits(shaderData); + stage.WriteTessellationData(shaderData); + if (layout.UsesRenderScale) + { + ulong scaledImages = 0; + var scaledCount = Math.Min(info.Images.Count, 64); + for (var index = 0; index < scaledCount; index++) + { + var identifier = descriptors.Images[index].ImageIdentifier; + if (identifier.IsValid && _imageCache.GetImage(identifier).IsScaled) + { + scaledImages |= 1UL << index; + } + } + + stage.WriteRenderScale(shaderData, scaledImages, RenderScalePolicy.Scale); + } + prepared.ShaderData = shaderData; if (layout.Find(DescriptorBindingKind.GlobalDataShare) is not null) { @@ -322,6 +453,12 @@ internal static unsafe partial class VulkanVideoPresenter return prepared; } + public void UpdateTessellationData(IPreparedBindings prepared, uint[] data) + { + var stage = (PreparedStageBindings)prepared; + (stage.Stage with { TessellationData = data }).WriteTessellationData(stage.ShaderData); + } + // Reject incompatible draw views before buffer writes or image transitions are recorded. private void ValidateDrawImageTypes(PreparedStageBindings prepared) { @@ -332,7 +469,7 @@ internal static unsafe partial class VulkanVideoPresenter foreach (var binding in prepared.Textures) { - if (binding.IsHostMovie || IsStaleImage(binding.ImageIdentifier, out var image) || image is null) + if (binding.IsHostMovie || !binding.IsResident || IsStaleImage(binding.ImageIdentifier, out var image) || image is null) { continue; } @@ -386,6 +523,13 @@ internal static unsafe partial class VulkanVideoPresenter } var size = ClampMappedSize(descriptor.Address, requested, prepared, index); + var resource = prepared.Resources.Info.Buffers[index]; + if (resource.Formatted && resource.Written) + { + // Image preservation can grow a buffer beyond its descriptor's range. + // All stages prepare here before any stage captures buffer handles. + _imageCache.InvalidateMemoryFromGpu(descriptor.Address, size); + } sources[index] = (descriptor, _bufferCache.FindBuffer(descriptor.Address, size)); } @@ -461,9 +605,46 @@ internal static unsafe partial class VulkanVideoPresenter BindBuffers(stage); ObtainDeviceAddressRanges(stage); BindImages(stage); + if (stage.Layout.Find(DescriptorBindingKind.RuntimeDescriptorTable) is not null) + { + BindRuntimeDescriptors(stage); + } + + BindFlattenedResourceTable(stage); _preparedTextures.Add(stage.Textures); } + // Only host movie planes own a staging buffer that must outlive the submission. Keeping + // every binding until GPU completion promoted each draw's bindings out of gen0. + private static TextureResource[] TexturesWithStaging(TextureResource[] textures) + { + var count = 0; + foreach (var texture in textures) + { + if (texture.StagingBuffer.Handle != 0) + { + count++; + } + } + + if (count == 0) + { + return []; + } + + var retained = new TextureResource[count]; + count = 0; + foreach (var texture in textures) + { + if (texture.StagingBuffer.Handle != 0) + { + retained[count++] = texture; + } + } + + return retained; + } + // Textures bound for the draw being prepared; cleared when its preparation closes. private readonly List _preparedTextures = new(); @@ -485,6 +666,12 @@ internal static unsafe partial class VulkanVideoPresenter return false; } + ImageLayout? IRenderHost.DepthAttachmentLayout(in DepthAttachmentState depth) + { + var view = depth.Target.Target.Request.View; + return _imageCache.GetImage(depth.Image).UniformLayout(new SubresourceRange(view.BaseLevel, view.LevelCount, view.BaseLayer, view.LayerCount)); + } + private BufferView NullStorageBuffer() => new(_bufferCache.GetBuffer(GuestBufferCache.NullBufferId).Handle, 0, NullStorageBufferBytes); // A storage buffer view on the cache buffer, aligned down with the adjustment carried in the memory offsets. @@ -494,9 +681,11 @@ internal static unsafe partial class VulkanVideoPresenter ShaderProgramInfo program, int slot, ResourceSlotIdentifier bufferIdentifier, - out uint memoryOffset) + out uint memoryOffset, + out uint tailPadding) { memoryOffset = 0; + tailPadding = 0; var address = descriptor.Address; var requested = descriptor.Footprint() ?? throw SubmissionScheduler.Fatal($"A storage buffer descriptor footprint overflows: buffer={slot} hash=0x{program.Hash:X16}."); if (address == 0 || requested == 0) @@ -515,18 +704,35 @@ internal static unsafe partial class VulkanVideoPresenter var (buffer, offset) = _bufferCache.ObtainBuffer(address, size, resource.Written, isTexelBuffer: resource.Formatted, bufferIdentifier); var alignedOffset = offset - offset % alignment; var adjustment = offset - alignedOffset; - if (adjustment % sizeof(uint) != 0 || adjustment >= MaxMemoryOffsetAdjustment || size > maxRange - adjustment) + // A buffer read only through byte-assembled accesses may start mid-dword. + if ((adjustment % sizeof(uint) != 0 && resource.DwordAddressed) || adjustment >= MaxMemoryOffsetAdjustment || size > maxRange - adjustment) { throw SubmissionScheduler.Fatal($"A storage buffer offset adjustment is unsupported: buffer={slot} adjustment={adjustment} hash=0x{program.Hash:X16}."); } memoryOffset = (uint)adjustment; - if (resource.Formatted && resource.Written) + var range = size + adjustment; + // The shader accesses uint[], and Vulkan's runtime array length truncates a partial + // final word. The hardware still reads and writes a word that starts inside the + // range, so the view is padded to whole words; the byte limit for byte and + // formatted elements travels in the unused high bits of the runtime stride. + if (program.Bindings!.UsesRuntimeBufferStrides) { - _imageCache.InvalidateMemoryFromGpu(address, size); + tailPadding = (uint)((0UL - range) & 3); + // Cache buffers span whole pages, so the padded word stays inside the VkBuffer. + if (range > maxRange - tailPadding || alignedOffset + range + tailPadding > buffer.Size) + { + throw SubmissionScheduler.Fatal($"A padded storage buffer exceeds the device limit or its cache buffer: buffer={slot} range=0x{range:X} offset=0x{alignedOffset:X} cache_size=0x{buffer.Size:X}."); + } + range += tailPadding; } - return new BufferView(buffer.Handle, alignedOffset, size + adjustment); + if (resource.Written && !resource.Formatted) + { + _imageCache.InvalidateMemoryCopiesFromGpu(address, size); + } + + return new BufferView(buffer.Handle, alignedOffset, range); } private BufferView UploadDwords(uint[] data, string label, ShaderProgramInfo program) @@ -556,6 +762,10 @@ internal static unsafe partial class VulkanVideoPresenter // Reset only packed buffer offsets; dispatch limits follow them in shader data. Array.Clear(shaderData, (int)layout.MemoryOffsetDword, (int)((layout.MemoryOffsetCount + 3) / 4)); var views = prepared.Descriptors.Buffers; + if (layout.UsesRuntimeBufferStrides) + { + Array.Clear(shaderData, (int)layout.BufferStrideDword, (int)layout.BufferStrideDwordCount); + } if (views.Length != info.Buffers.Count) { Scratch.Buffers.Return(views); @@ -565,24 +775,78 @@ internal static unsafe partial class VulkanVideoPresenter for (var index = 0; index < views.Length; index++) { var (descriptor, bufferIdentifier) = prepared.BufferSources[index]; - views[index] = BindStorageBuffer(in descriptor, info.Buffers[index], program, index, bufferIdentifier, out var memoryOffset); + views[index] = BindStorageBuffer(in descriptor, info.Buffers[index], program, index, bufferIdentifier, out var memoryOffset, out var tailPadding); var dword = layout.MemoryOffsetDword + (uint)index / 4; var shift = ((uint)index % 4) * 8; shaderData[dword] |= memoryOffset << (int)shift; + if (layout.UsesRuntimeBufferStrides) + { + var stride = descriptor.Stride | (tailPadding << 14); + shaderData[layout.BufferStrideDword + (uint)index / 2] |= stride << ((index % 2) * 16); + } } prepared.Descriptors.Buffers = views; - if (layout.Find(DescriptorBindingKind.FlattenedResourceTable) is not null) - { - prepared.Descriptors.FlattenedTable = UploadDwords(snapshot.FlattenedResourceTable, "flattened resource table", program); - } - if (layout.Find(DescriptorBindingKind.ShaderData) is not null) { prepared.Descriptors.ShaderData = UploadDwords(shaderData, "shader data", program); } } + private void BindFlattenedResourceTable(PreparedStageBindings prepared) + { + var layout = prepared.Layout; + if (layout.Find(DescriptorBindingKind.FlattenedResourceTable) is null) + { + return; + } + + var program = prepared.Program; + var snapshot = prepared.Stage.Resources; + if (!layout.UsesBindlessImages) + { + prepared.Descriptors.FlattenedTable = UploadDwords(snapshot.FlattenedResourceTable, "flattened resource table", program); + return; + } + + var slotCount = (int)BindingLayout.ImageSlotTableDwordCount(prepared.Resources.Info); + var table = new uint[checked(slotCount + snapshot.FlattenedResourceTable.Length)]; + var slot = 0; + foreach (var binding in layout.Descriptors) + { + if (ImageDescriptorBinding.ResourceClass(binding.Kind) == ShaderCompiler.Resources.ImageResourceClass.None) + { + continue; + } + + var occurrences = new uint[prepared.Descriptors.Images.Length]; + foreach (var resource in binding.Resources) + { + var texture = prepared.Descriptors.Images[resource]; + var occurrence = occurrences[resource]++; + if (!texture.IsResident) + { + table[slot++] = 0; + continue; + } + + var view = texture.MipViews.Length == 0 + ? texture.View + : occurrence < (uint)texture.MipViews.Length ? texture.MipViews[occurrence] : default; + if (view.Handle == 0) + { + throw SubmissionScheduler.Fatal($"A bindless image has no view: image={resource} hash=0x{program.Hash:X16}."); + } + + table[slot++] = _bindlessImageHeap!.GetOrCreateSlot( + binding.Kind, prepared.Stage.Resources.Images[(int)resource], view, texture.Layout); + } + } + + snapshot.FlattenedResourceTable.AsSpan().CopyTo(table.AsSpan(slotCount)); + prepared.Descriptors.FlattenedTable = UploadDwords(table, "bindless image slot table", program); + } + // Device-address reads need persistent page-table entries before the shader runs. private void ObtainDeviceAddressRanges(PreparedStageBindings prepared) { @@ -612,15 +876,17 @@ internal static unsafe partial class VulkanVideoPresenter } var size = ClampMappedSize(range.Base, range.Size); + if (range.Written) { _ = _bufferCache.ObtainBuffer(range.Base, size, isWritten: true); + _imageCache.InvalidateMemoryCopiesFromGpu(range.Base, size); } else { // Stream buffers do not populate the device-address page table. _ = _bufferCache.FindBuffer(range.Base, size); - _bufferCache.SynchronizeBuffersInRange(range.Base, size); + _bufferCache.SynchronizeDeviceAddressRange(range.Base, size); } } } @@ -635,7 +901,7 @@ internal static unsafe partial class VulkanVideoPresenter for (var index = 0; index < images.Length; index++) { var binding = images[index]; - if (binding.IsHostMovie) + if (binding.IsHostMovie || !binding.IsResident) { continue; } @@ -654,7 +920,7 @@ internal static unsafe partial class VulkanVideoPresenter for (var index = 0; index < images.Length; index++) { var binding = images[index]; - if (binding.IsHostMovie) + if (binding.IsHostMovie || !binding.IsResident) { continue; } @@ -698,6 +964,12 @@ internal static unsafe partial class VulkanVideoPresenter binding.MipViews = []; } + var descriptor = new TextureDescriptorWords(snapshot.Images[index]); + if (descriptor.MetadataCompress) + { + _imageCache.ApplyPendingDccClear(binding.ImageIdentifier, descriptor.MetadataAddress << 8); + } + image.Uses.Storage |= binding.IsStorage; image.Uses.Texture |= !binding.IsStorage; binding.CachedImage = image; @@ -751,8 +1023,19 @@ internal static unsafe partial class VulkanVideoPresenter using var profileScope = RenderPhaseProfile.MeasureDetail(RenderPhaseProfile.Phase.DescriptorCommit); var preparation = RequirePreparation(); var entry = RequirePipelineEntry(in pipeline); + foreach (var prepared in stages) + { + var stage = (PreparedStageBindings)prepared; + if (stage.Layout.Find(DescriptorBindingKind.RuntimeDescriptorTable) is not null) + CommitRuntimeDescriptorBuffers(stage); + } var command = BeginBatchedGuestCommands(); _commandBuffer = command; + if (entry.UsesBindlessImages) + { + var globalSet = _bindlessImageHeap!.Set; + _vk.CmdBindDescriptorSets(command, bindPoint, entry.Layout, 0, 1, &globalSet, 0, null); + } var writeCount = 0; var bufferCount = 0; var imageCount = 0; @@ -762,7 +1045,7 @@ internal static unsafe partial class VulkanVideoPresenter { var stage = (PreparedStageBindings)prepared; var stageFlag = DescriptorWriter.ShaderStageFlag(StageOf(stage.Program)); - if ((bindPoint == PipelineBindPoint.Graphics && (stageFlag & (ShaderStageFlags.VertexBit | ShaderStageFlags.FragmentBit)) == 0) || + if ((bindPoint == PipelineBindPoint.Graphics && (stageFlag & (ShaderStageFlags.VertexBit | ShaderStageFlags.FragmentBit | ShaderStageFlags.TessellationEvaluationBit)) == 0) || (bindPoint == PipelineBindPoint.Compute && stageFlag != ShaderStageFlags.ComputeBit)) { throw SubmissionScheduler.Fatal($"A stage does not belong to the bind point: stage={stage.Program.Stage} bindPoint={bindPoint}."); @@ -770,6 +1053,11 @@ internal static unsafe partial class VulkanVideoPresenter foreach (var binding in stage.Layout.Descriptors) { + if (stage.Layout.UsesBindlessImages && ImageDescriptorBinding.ResourceClass(binding.Kind) != ShaderCompiler.Resources.ImageResourceClass.None) + { + continue; + } + writeCount++; var count = (int)DescriptorWriter.DescriptorCount(binding); if (ImageDescriptorBinding.ResourceClass(binding.Kind) != ShaderCompiler.Resources.ImageResourceClass.None || binding.Kind == DescriptorBindingKind.Samplers) @@ -822,7 +1110,7 @@ internal static unsafe partial class VulkanVideoPresenter var bufferIndex = 0; var imageIndex = 0; var writeIndex = 0; - var uploadStage = bindPoint == PipelineBindPoint.Compute ? PipelineStageFlags.ComputeShaderBit : PipelineStageFlags.FragmentShaderBit; + var uploadStage = bindPoint == PipelineBindPoint.Compute ? PipelineStageFlags.ComputeShaderBit : PipelineStageFlags.AllGraphicsBit; TextureResource[] textures = textureCount == 0 ? [] : new TextureResource[textureCount]; var textureIndex = 0; fixed (DescriptorBufferInfo* bufferInfoPointer = bufferInfos) @@ -864,6 +1152,11 @@ internal static unsafe partial class VulkanVideoPresenter occurrences.Clear(); foreach (var binding in stage.Layout.Descriptors) { + if (stage.Layout.UsesBindlessImages && ImageDescriptorBinding.ResourceClass(binding.Kind) != ShaderCompiler.Resources.ImageResourceClass.None) + { + continue; + } + var write = new WriteDescriptorSet { SType = StructureType.WriteDescriptorSet, @@ -908,11 +1201,15 @@ internal static unsafe partial class VulkanVideoPresenter case DescriptorBindingKind.FlattenedResourceTable: case DescriptorBindingKind.ShaderData: case DescriptorBindingKind.GlobalDataShare: + case DescriptorBindingKind.RuntimeDescriptorTable: + case DescriptorBindingKind.RuntimeDescriptorMisses: { var view = binding.Kind switch { DescriptorBindingKind.FlattenedResourceTable => descriptors.FlattenedTable, DescriptorBindingKind.ShaderData => descriptors.ShaderData, + DescriptorBindingKind.RuntimeDescriptorTable => descriptors.RuntimeTable, + DescriptorBindingKind.RuntimeDescriptorMisses => descriptors.RuntimeMisses, _ => descriptors.GlobalDataShare, }; if (view.Buffer.Handle == 0) @@ -955,7 +1252,7 @@ internal static unsafe partial class VulkanVideoPresenter writes[writeIndex++] = write; } - for (var index = 0; index < descriptors.Images.Length; index++) + for (var index = 0; index < descriptors.Images.Length && !stage.Layout.UsesBindlessImages; index++) { var expected = descriptors.Images[index].MipViews.Length == 0 ? 1u : (uint)descriptors.Images[index].MipViews.Length; if (occurrences[index] != expected) @@ -982,7 +1279,8 @@ internal static unsafe partial class VulkanVideoPresenter if (hasPushData) { - var pushStages = bindPoint == PipelineBindPoint.Graphics ? ShaderStageFlags.VertexBit | ShaderStageFlags.FragmentBit : ShaderStageFlags.ComputeBit; + var pushStages = bindPoint == PipelineBindPoint.Graphics ? ShaderStageFlags.VertexBit | ShaderStageFlags.FragmentBit | + (entry.Description?.Tessellation is null ? 0 : ShaderStageFlags.TessellationControlBit | ShaderStageFlags.TessellationEvaluationBit) : ShaderStageFlags.ComputeBit; _vk.CmdPushConstants(command, entry.Layout, pushStages, 0, PushData.ByteSize, pushData); } @@ -992,7 +1290,7 @@ internal static unsafe partial class VulkanVideoPresenter { if (entry.UsesPushDescriptors) { - _pushDescriptorApi.CmdPushDescriptorSet(command, bindPoint, entry.Layout, 0, (uint)writeIndex, writePointer); + _pushDescriptorApi.CmdPushDescriptorSet(command, bindPoint, entry.Layout, entry.UsesBindlessImages ? 1u : 0u, (uint)writeIndex, writePointer); ShaderCacheCounters.CountPushSet(); } else @@ -1004,7 +1302,8 @@ internal static unsafe partial class VulkanVideoPresenter } _vk.UpdateDescriptorSets(_device, (uint)writeIndex, writePointer, 0, null); - _vk.CmdBindDescriptorSets(command, bindPoint, entry.Layout, 0, 1, &set, 0, null); + var setIndex = entry.UsesBindlessImages ? 1u : 0u; + _vk.CmdBindDescriptorSets(command, bindPoint, entry.Layout, setIndex, 1, &set, 0, null); ShaderCacheCounters.CountHeapSet(); } } @@ -1014,7 +1313,7 @@ internal static unsafe partial class VulkanVideoPresenter _batchResources.Add(new SubmissionUploadResources { DebugName = bindPoint == PipelineBindPoint.Compute ? "SharpEmu dispatch" : "SharpEmu draw", - Textures = textures, + Textures = TexturesWithStaging(textures), FeedbackSnapshots = preparation.FeedbackSnapshots?.ToArray() ?? [], OverflowBuffers = preparation.OverflowBuffers.Count == 0 ? null : preparation.OverflowBuffers.ToArray(), }); diff --git a/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.Device.Setup.cs b/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.Device.Setup.cs index 9d0d1d0..b82bbb5 100644 --- a/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.Device.Setup.cs +++ b/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.Device.Setup.cs @@ -8,6 +8,7 @@ using SharpEmu.Libs.Gpu.Buffers; using SharpEmu.Libs.Gpu.Pipelines; using SharpEmu.Libs.Gpu.Rendering; using SharpEmu.Libs.Gpu.Scheduling; +using SharpEmu.Libs.Gpu.Vulkan; using Silk.NET.Core; using Silk.NET.Core.Native; using Silk.NET.Vulkan; @@ -44,6 +45,7 @@ internal static unsafe partial class VulkanVideoPresenter private uint _maxComputeWorkGroupInvocations; private ulong _minStorageBufferOffsetAlignment = 1; private bool _supportsIndependentBlend; + private bool _supportsDualSourceBlend; private bool _supportsDepthBiasClamp; private bool _supportsShaderSignedZeroInfNanPreserveFloat32; private uint _maxColorAttachments; @@ -57,9 +59,9 @@ internal static unsafe partial class VulkanVideoPresenter private const long PipelineCacheCheckpointIntervalMs = 300_000; private const long PipelineCacheRetryMs = 1_000; private long _pipelineCacheRetryTick; - private const ulong MaxPipelineCacheBytes = 256UL * 1024 * 1024; - private HashSet? _resetPipelineCachePaths; + private const ulong MaxPipelineCacheBytes = ulong.MaxValue; private nuint _pipelineCacheSavedBytes; + private HashSet? _resetPipelineCachePaths; private Queue _queue; private uint _queueFamilyIndex; @@ -89,6 +91,8 @@ internal static unsafe partial class VulkanVideoPresenter CreateSwapchain(); CreateCommandResources(); CreateGuestDrawResources(); + ProbeNativeHalfConversion(); + ProbeZeroOutOfBoundsReads(); _vulkanReady = true; AttachGuestGpuMemory(); Console.Error.WriteLine( @@ -584,10 +588,15 @@ internal static unsafe partial class VulkanVideoPresenter { SType = StructureType.PhysicalDevicePushDescriptorPropertiesKhr, }; + var descriptorIndexingProperties = new PhysicalDeviceDescriptorIndexingProperties + { + SType = StructureType.PhysicalDeviceDescriptorIndexingProperties, + PNext = &pushDescriptorProperties, + }; var floatControls = new PhysicalDeviceFloatControlsProperties { SType = StructureType.PhysicalDeviceFloatControlsProperties, - PNext = &pushDescriptorProperties, + PNext = &descriptorIndexingProperties, }; var subgroupSizeControl = new PhysicalDeviceSubgroupSizeControlProperties { @@ -614,6 +623,21 @@ internal static unsafe partial class VulkanVideoPresenter subgroupSizeControl.MaxSubgroupSize >= RdnaSubgroupSize && (subgroupSizeControl.RequiredSubgroupSizeStages & ShaderStageFlags.ComputeBit) != 0; _maxComputeWorkgroupSubgroups = subgroupSizeControl.MaxComputeWorkgroupSubgroups; + + _maxPerStageSampledImages = properties.Limits.MaxPerStageDescriptorSampledImages; + _maxPerStageStorageImages = properties.Limits.MaxPerStageDescriptorStorageImages; + _maxPerStageStorageBuffers = properties.Limits.MaxPerStageDescriptorStorageBuffers; + _maxPerStageUpdateAfterBindSampledImages = descriptorIndexingProperties.MaxPerStageDescriptorUpdateAfterBindSampledImages; + _maxPerStageUpdateAfterBindStorageImages = descriptorIndexingProperties.MaxPerStageDescriptorUpdateAfterBindStorageImages; + _maxUpdateAfterBindSampledImages = descriptorIndexingProperties.MaxDescriptorSetUpdateAfterBindSampledImages; + _maxUpdateAfterBindStorageImages = descriptorIndexingProperties.MaxDescriptorSetUpdateAfterBindStorageImages; + _maxUpdateAfterBindDescriptors = descriptorIndexingProperties.MaxUpdateAfterBindDescriptorsInAllPools; + _maxPerStageSamplers = properties.Limits.MaxPerStageDescriptorSamplers; + _maxPerStageUpdateAfterBindSamplers = descriptorIndexingProperties.MaxPerStageDescriptorUpdateAfterBindSamplers; + _maxUpdateAfterBindSamplers = descriptorIndexingProperties.MaxDescriptorSetUpdateAfterBindSamplers; + Console.Error.WriteLine( + $"[LOADER][INFO] Vulkan bindless limits sampled={_maxUpdateAfterBindSampledImages} " + + $"storage={_maxUpdateAfterBindStorageImages} total={_maxUpdateAfterBindDescriptors}"); // MoltenVK reports a zero OpArrayLength for storage buffers bound through push // descriptors, which turns every bounds-checked load into zero and drops every store. _maxPushDescriptors = Gpu.Vulkan.VulkanPushDescriptorPolicy.UsableCount(_vk, _physicalDevice, pushDescriptorProperties.MaxPushDescriptors); @@ -695,6 +719,7 @@ internal static unsafe partial class VulkanVideoPresenter } uint? compute = null; + uint? general = null; for (uint index = 0; index < count; index++) { var flags = families[index].QueueFlags; @@ -712,21 +737,168 @@ internal static unsafe partial class VulkanVideoPresenter { compute ??= index; } + + // MoltenVK exposes only general families, one queue each, every one its own + // Metal command queue: a second one still runs readbacks beside the main queue. + if ((flags & (QueueFlags.TransferBit | QueueFlags.GraphicsBit | QueueFlags.ComputeBit)) != 0) + { + general ??= index; + } } - return compute; + return compute ?? general; + } + + // Measures GLSL UnpackHalf2x16 / PackHalf2x16 against the translator's own f16 conversion + // once per device, before any guest shader is compiled, so the answer is constant for the + // process. The spec does not promise it: MoltenVK reports RTE rounding and signed-zero / + // Inf / NaN preservation for f16 yet not denorm preservation, so only the device can say. + // Never fatal - anything that goes wrong leaves the per-access checks in place. The probe + // reads out of range on purpose, so it only runs where the device promised a zero: without + // robustBufferAccess2 an out-of-range read is undefined and may fault the device. + private void ProbeZeroOutOfBoundsReads() + { + if (Interlocked.Exchange(ref _zeroOutOfBoundsReadsProbed, 1) != 0) + { + return; + } + + var forced = ZeroOutOfBoundsReadProbe.ReadOverride(); + if (forced is not null) + { + Volatile.Write(ref _zeroOutOfBoundsReads, forced.Value ? 1 : 0); + Console.Error.WriteLine( + $"[VK][OOB] out-of-range buffer reads zero={forced.Value} mismatches=unprobed tested=0 " + + "(SHARPEMU_ZERO_OOB_READS)"); + return; + } + + if (!_robustBufferAccess2Enabled) + { + Console.Error.WriteLine( + "[VK][OOB] out-of-range buffer reads zero=False mismatches=unprobed tested=0 " + + "note=robustBufferAccess2 is not enabled on this device"); + return; + } + + var result = ZeroOutOfBoundsReadProbe.Run(_deviceInfo, _scheduler, CurrentRecordingBuffer); + Volatile.Write(ref _zeroOutOfBoundsReads, result.Zeroed ? 1 : 0); + Console.Error.WriteLine( + $"[VK][OOB] out-of-range buffer reads zero={result.Zeroed} " + + $"mismatches={result.Mismatches} tested={result.Tested}" + + (result.Note is null ? string.Empty : $" note={result.Note}")); + } + + // Never fatal - anything that goes wrong leaves the exact emulation in place. + private void ProbeNativeHalfConversion() + { + if (Interlocked.Exchange(ref _nativeHalfConversionProbed, 1) != 0) + { + return; + } + + var forced = NativeHalfConversionProbe.ReadOverride(); + if (forced is not null) + { + Volatile.Write(ref _nativeHalfConversionExact, forced.Value ? 1 : 0); + Console.Error.WriteLine( + $"[VK][F16] native half conversion exact={forced.Value} mismatches=unprobed tested=0 " + + "(SHARPEMU_NATIVE_HALF)"); + return; + } + + var result = NativeHalfConversionProbe.Run(_deviceInfo, _scheduler, CurrentRecordingBuffer); + Volatile.Write(ref _nativeHalfConversionExact, result.Exact ? 1 : 0); + Console.Error.WriteLine( + $"[VK][F16] native half conversion exact={result.Exact} " + + $"mismatches={result.Mismatches} tested={result.Tested}" + + (result.Note is null ? string.Empty : $" note={result.Note}") + + (result.Detail is null ? string.Empty : $" first={result.Detail}")); } private bool _supportsFragmentShaderBarycentric; private bool _supportsPerVertexPixelInputs; private const string FragmentShaderBarycentricExtensionName = "VK_KHR_fragment_shader_barycentric"; + private const string DeviceFaultExtensionName = "VK_EXT_device_fault"; + private const string MemoryBudgetExtensionName = "VK_EXT_memory_budget"; + private bool _memoryBudgetEnabled; private const string Maintenance5ExtensionName = "VK_KHR_maintenance5"; private const string ImageViewMinLodExtensionName = "VK_EXT_image_view_min_lod"; private const string FillRectangleExtensionName = "VK_NV_fill_rectangle"; private bool _supportsImageViewMinLod; + private bool _supportsNativeTwoSampleMixed; + private bool _supportsVariableSampleLocations; private void CreateDevice() { + var supportedTwoSampleCombination = false; + if (IsDeviceExtensionAvailable("VK_NV_coverage_reduction_mode")) + { + var function = _vk.GetInstanceProcAddr(_instance, "vkGetPhysicalDeviceSupportedFramebufferMixedSamplesCombinationsNV"); + if (function.Handle != null) + { + var query = (delegate* unmanaged)function.Handle; + uint count = 0; + var result = query(_physicalDevice, &count, null); + if (result == Result.Success && count <= 4096) + { + var combinations = new FramebufferMixedSamplesCombinationNV[count]; + for (var index = 0; index < combinations.Length; index++) + combinations[index].SType = StructureType.FramebufferMixedSamplesCombinationNV; + fixed (FramebufferMixedSamplesCombinationNV* pointer = combinations) + { + var capacity = count; + result = query(_physicalDevice, &count, pointer); + if (result == Result.Success && count <= capacity) + for (var index = 0; index < count; index++) + { + var combination = combinations[index]; + supportedTwoSampleCombination |= combination.CoverageReductionMode == CoverageReductionModeNV.TruncateNV && + combination.RasterizationSamples == SampleCountFlags.Count2Bit && + (combination.DepthStencilSamples & SampleCountFlags.Count2Bit) != 0 && + (combination.ColorSamples & SampleCountFlags.Count1Bit) != 0; + } + } + } + } + } + var nativeMixed = supportedTwoSampleCombination && + IsDeviceExtensionAvailable("VK_NV_framebuffer_mixed_samples") && + IsDeviceExtensionAvailable("VK_EXT_sample_locations"); + var nativeMixedFeatures = new PhysicalDeviceCoverageReductionModeFeaturesNV + { + SType = StructureType.PhysicalDeviceCoverageReductionModeFeaturesNV, + }; + if (nativeMixed) + { + var coverageQuery = new PhysicalDeviceFeatures2 + { + SType = StructureType.PhysicalDeviceFeatures2, + PNext = &nativeMixedFeatures, + }; + _vk.GetPhysicalDeviceFeatures2(_physicalDevice, &coverageQuery); + nativeMixed &= nativeMixedFeatures.CoverageReductionMode; + var sampleLocationProperties = new PhysicalDeviceSampleLocationsPropertiesEXT + { + SType = StructureType.PhysicalDeviceSampleLocationsPropertiesExt, + }; + var samplePropertyQuery = new PhysicalDeviceProperties2 { SType = StructureType.PhysicalDeviceProperties2, PNext = &sampleLocationProperties }; + _vk.GetPhysicalDeviceProperties2(_physicalDevice, &samplePropertyQuery); + _supportsVariableSampleLocations = sampleLocationProperties.VariableSampleLocations; + var sampleGridProperties = new MultisamplePropertiesEXT { SType = StructureType.MultisamplePropertiesExt }; + var sampleGridFunction = _vk.GetInstanceProcAddr(_instance, "vkGetPhysicalDeviceMultisamplePropertiesEXT"); + if (sampleGridFunction.Handle != null) + ((delegate* unmanaged)sampleGridFunction.Handle)(_physicalDevice, SampleCountFlags.Count2Bit, &sampleGridProperties); + else + nativeMixed = false; + if ((sampleLocationProperties.SampleLocationSampleCounts & SampleCountFlags.Count2Bit) == 0 || + sampleGridProperties.MaxSampleLocationGridSize.Width < 2 || sampleGridProperties.MaxSampleLocationGridSize.Height < 2 || + sampleLocationProperties.SampleLocationSubPixelBits < 4 || + sampleLocationProperties.SampleLocationCoordinateRange[0] > 0f || + sampleLocationProperties.SampleLocationCoordinateRange[1] < 15f / 16f) + nativeMixed = false; + } + _supportsNativeTwoSampleMixed = nativeMixed; var priority = 1.0f; var queueInfos = stackalloc DeviceQueueCreateInfo[2]; queueInfos[0] = new DeviceQueueCreateInfo @@ -751,14 +923,17 @@ internal static unsafe partial class VulkanVideoPresenter } _vk.GetPhysicalDeviceFeatures(_physicalDevice, out var supportedFeatures); _supportsIndependentBlend = supportedFeatures.IndependentBlend; + _supportsDualSourceBlend = supportedFeatures.DualSrcBlend; _supportsDepthBiasClamp = supportedFeatures.DepthBiasClamp; _supportsDepthBounds = supportedFeatures.DepthBounds; _supportsShaderClipDistance = supportedFeatures.ShaderClipDistance; _supportsFillRectangle = IsDeviceExtensionAvailable(FillRectangleExtensionName); var enabledFeatures = new PhysicalDeviceFeatures { + TessellationShader = supportedFeatures.TessellationShader, DepthBounds = supportedFeatures.DepthBounds, IndependentBlend = supportedFeatures.IndependentBlend, + DualSrcBlend = supportedFeatures.DualSrcBlend, VertexPipelineStoresAndAtomics = supportedFeatures.VertexPipelineStoresAndAtomics, FragmentStoresAndAtomics = supportedFeatures.FragmentStoresAndAtomics, ShaderInt64 = supportedFeatures.ShaderInt64, @@ -930,10 +1105,15 @@ internal static unsafe partial class VulkanVideoPresenter SType = StructureType.PhysicalDeviceShaderAtomicInt64Features, PNext = &addressFeatures, }; + var descriptorIndexingQuery = new PhysicalDeviceDescriptorIndexingFeatures + { + SType = StructureType.PhysicalDeviceDescriptorIndexingFeatures, + PNext = &atomicInt64Features, + }; var featuresQuery = new PhysicalDeviceFeatures2 { SType = StructureType.PhysicalDeviceFeatures2, - PNext = &atomicInt64Features, + PNext = &descriptorIndexingQuery, }; _vk.GetPhysicalDeviceFeatures2(_physicalDevice, &featuresQuery); var supportsTimelineSemaphore = timelineSemaphoreFeatures.TimelineSemaphore; @@ -946,6 +1126,30 @@ internal static unsafe partial class VulkanVideoPresenter var supportsRobustness2 = supportsRobustImageAccess2 || supportsNullDescriptor; _canRequireComputeSubgroup32 &= vulkan13Features.SubgroupSizeControl; SetSharedInt64AtomicsCapability(supportsSharedInt64Atomics); + var supportsExactFloat16 = VulkanFloat16Support.SupportsExactConversions(_vk, _physicalDevice); + SetExactFloat16ConversionsCapability(supportsExactFloat16); + var supportsNonUniformImageIndexing = descriptorIndexingQuery.ShaderSampledImageArrayNonUniformIndexing && + descriptorIndexingQuery.ShaderStorageImageArrayNonUniformIndexing; + SetNonUniformImageIndexingCapability(supportsNonUniformImageIndexing); + var supportsBindlessImageHeap = supportsNonUniformImageIndexing && + descriptorIndexingQuery.RuntimeDescriptorArray && + descriptorIndexingQuery.DescriptorBindingPartiallyBound && + descriptorIndexingQuery.DescriptorBindingSampledImageUpdateAfterBind && + descriptorIndexingQuery.DescriptorBindingStorageImageUpdateAfterBind && + descriptorIndexingQuery.DescriptorBindingUpdateUnusedWhilePending && + supportsNullDescriptor; + SetBindlessImageHeapCapability(supportsBindlessImageHeap); + var descriptorIndexingFeatures = new PhysicalDeviceDescriptorIndexingFeatures + { + SType = StructureType.PhysicalDeviceDescriptorIndexingFeatures, + ShaderSampledImageArrayNonUniformIndexing = supportsNonUniformImageIndexing, + ShaderStorageImageArrayNonUniformIndexing = supportsNonUniformImageIndexing, + RuntimeDescriptorArray = supportsBindlessImageHeap, + DescriptorBindingPartiallyBound = supportsBindlessImageHeap, + DescriptorBindingSampledImageUpdateAfterBind = supportsBindlessImageHeap, + DescriptorBindingStorageImageUpdateAfterBind = supportsBindlessImageHeap, + DescriptorBindingUpdateUnusedWhilePending = supportsBindlessImageHeap, + }; if (!supportsSharedInt64Atomics) { Console.Error.WriteLine( @@ -989,15 +1193,33 @@ internal static unsafe partial class VulkanVideoPresenter var depthClipEnableExtension = (byte*)SilkMarshal.StringToPtr(DepthClipEnableExtensionName); var barycentricExtension = (byte*)SilkMarshal.StringToPtr(FragmentShaderBarycentricExtensionName); var viewportIndexLayerExtension = (byte*)SilkMarshal.StringToPtr("VK_EXT_shader_viewport_index_layer"); + var deviceFaultExtension = (byte*)SilkMarshal.StringToPtr(DeviceFaultExtensionName); + var memoryBudgetExtension = (byte*)SilkMarshal.StringToPtr(MemoryBudgetExtensionName); + var supportsDeviceFault = IsDeviceExtensionAvailable(DeviceFaultExtensionName); + _memoryBudgetEnabled = IsDeviceExtensionAvailable(MemoryBudgetExtensionName); + var deviceFaultFeatures = new PhysicalDeviceFaultFeaturesEXT { SType = StructureType.PhysicalDeviceFaultFeaturesExt }; var maintenance5Extension = (byte*)SilkMarshal.StringToPtr(Maintenance5ExtensionName); var imageViewMinLodExtension = (byte*)SilkMarshal.StringToPtr(ImageViewMinLodExtensionName); var fillRectangleExtension = (byte*)SilkMarshal.StringToPtr(FillRectangleExtensionName); + var nativeMixedExtension = (byte*)SilkMarshal.StringToPtr("VK_NV_framebuffer_mixed_samples"); + var coverageReductionExtension = (byte*)SilkMarshal.StringToPtr("VK_NV_coverage_reduction_mode"); + var sampleLocationsExtension = (byte*)SilkMarshal.StringToPtr("VK_EXT_sample_locations"); + var postDepthCoverageExtension = (byte*)SilkMarshal.StringToPtr("VK_EXT_post_depth_coverage"); try { - var extensions = stackalloc byte*[15]; + var extensions = stackalloc byte*[24]; var extensionCount = 0u; extensions[extensionCount++] = swapchainExtension; extensions[extensionCount++] = pushDescriptorExtension; + _postDepthCoverageEnabled = IsDeviceExtensionAvailable("VK_EXT_post_depth_coverage"); + if (_postDepthCoverageEnabled) + extensions[extensionCount++] = postDepthCoverageExtension; + if (nativeMixed) + { + extensions[extensionCount++] = nativeMixedExtension; + extensions[extensionCount++] = coverageReductionExtension; + extensions[extensionCount++] = sampleLocationsExtension; + } if (IsDeviceExtensionAvailable("VK_EXT_shader_viewport_index_layer")) { extensions[extensionCount++] = viewportIndexLayerExtension; @@ -1046,6 +1268,17 @@ internal static unsafe partial class VulkanVideoPresenter extensions[extensionCount++] = robustness2Extension; } + // Device-loss reports name the faulting address when the driver can. + if (supportsDeviceFault) + { + extensions[extensionCount++] = deviceFaultExtension; + } + + if (_memoryBudgetEnabled) + { + extensions[extensionCount++] = memoryBudgetExtension; + } + if (IsDeviceExtensionAvailable(PortabilitySubsetExtensionName)) { // The spec requires enabling this when the (MoltenVK) @@ -1057,6 +1290,8 @@ internal static unsafe partial class VulkanVideoPresenter maintenance8Features.PNext = null; robustness2Features.RobustBufferAccess2 = supportsRobustBufferAccess2 && supportedFeatures.RobustBufferAccess; + // The feature only reaches the device when the robustness2 struct is chained. + _robustBufferAccess2Enabled = robustness2Features.RobustBufferAccess2 && supportsRobustness2; robustness2Features.RobustImageAccess2 = supportsRobustImageAccess2; robustness2Features.NullDescriptor = supportsNullDescriptor; robustness2Features.PNext = supportsMaintenance8 ? &maintenance8Features : null; @@ -1083,6 +1318,11 @@ internal static unsafe partial class VulkanVideoPresenter PNext = &timelineSemaphoreFeatures, }; void* renderingChain = &addressFeatures; + if (nativeMixed) + { + nativeMixedFeatures.PNext = renderingChain; + renderingChain = &nativeMixedFeatures; + } if (supportsSharedInt64Atomics) { atomicInt64Features = new PhysicalDeviceShaderAtomicInt64Features @@ -1094,6 +1334,17 @@ internal static unsafe partial class VulkanVideoPresenter renderingChain = &atomicInt64Features; } + var float16Features = new PhysicalDeviceShaderFloat16Int8Features + { + SType = StructureType.PhysicalDeviceShaderFloat16Int8Features, + ShaderFloat16 = true, + }; + if (supportsExactFloat16) + { + float16Features.PNext = renderingChain; + renderingChain = &float16Features; + } + if (_supportsFragmentShaderBarycentric) { barycentricFeatures.PNext = renderingChain; @@ -1152,6 +1403,16 @@ internal static unsafe partial class VulkanVideoPresenter renderingChain = &colorWriteEnableFeatures; } + descriptorIndexingFeatures.PNext = renderingChain; + renderingChain = &descriptorIndexingFeatures; + + if (supportsDeviceFault) + { + deviceFaultFeatures.DeviceFault = true; + deviceFaultFeatures.PNext = renderingChain; + renderingChain = &deviceFaultFeatures; + } + vulkan13Features = new PhysicalDeviceVulkan13Features { SType = StructureType.PhysicalDeviceVulkan13Features, @@ -1180,6 +1441,10 @@ internal static unsafe partial class VulkanVideoPresenter } finally { + SilkMarshal.Free((nint)nativeMixedExtension); + SilkMarshal.Free((nint)coverageReductionExtension); + SilkMarshal.Free((nint)sampleLocationsExtension); + SilkMarshal.Free((nint)postDepthCoverageExtension); SilkMarshal.Free((nint)swapchainExtension); SilkMarshal.Free((nint)maintenance8Extension); SilkMarshal.Free((nint)maintenance5Extension); @@ -1193,10 +1458,13 @@ internal static unsafe partial class VulkanVideoPresenter SilkMarshal.Free((nint)pushDescriptorExtension); SilkMarshal.Free((nint)barycentricExtension); SilkMarshal.Free((nint)viewportIndexLayerExtension); + SilkMarshal.Free((nint)deviceFaultExtension); + SilkMarshal.Free((nint)memoryBudgetExtension); } _vk.GetDeviceQueue(_device, _queueFamilyIndex, 0, out _queue); - _deviceInfo = new GpuDeviceInfo(_vk, _physicalDevice, _device) { ImageViewMinLodSupported = _supportsImageViewMinLod }; + _deviceInfo = new GpuDeviceInfo(_vk, _physicalDevice, _device, _memoryBudgetEnabled) { ImageViewMinLodSupported = _supportsImageViewMinLod, CustomTwoSampleLocationsSupported = _supportsNativeTwoSampleMixed }; + _deviceInfo.NameObject = SetDebugName; if (_readbackQueueFamilyIndex is { } readbackQueueFamily) { _vk.GetDeviceQueue(_device, readbackQueueFamily, 0, out _readbackQueue); @@ -1246,16 +1514,19 @@ internal static unsafe partial class VulkanVideoPresenter RemoveSingleFilePipelineCache(singleFilePath); } } - byte[] initialData = []; + byte* initialData = null; + ulong initialLength = 0; try { if (_pipelineCachePath is not null && File.Exists(_pipelineCachePath)) { - if (!PipelineCacheSignature.TryUnwrap(DriverCacheSignature(), File.ReadAllBytes(_pipelineCachePath), out initialData)) + using var file = File.OpenRead(_pipelineCachePath); + if (!PipelineCacheSignature.TryReadFrom(file, DriverCacheSignature(), out initialData, out initialLength)) { Console.Error.WriteLine( $"[LOADER][INFO] Vulkan pipeline cache invalidated: path={_pipelineCachePath} reason=signature-or-hash-mismatch"); - initialData = []; + initialData = null; + initialLength = 0; } } } @@ -1265,12 +1536,19 @@ internal static unsafe partial class VulkanVideoPresenter $"[LOADER][WARN] Vulkan pipeline cache read failed: {exception.Message}"); } - var result = TryCreatePipelineCache(initialData, out _pipelineCache); - if (result != Result.Success && initialData.Length != 0) + Result result; + try { - Console.Error.WriteLine( - $"[LOADER][WARN] Vulkan pipeline cache rejected ({result}); rebuilding it."); - result = TryCreatePipelineCache([], out _pipelineCache); + result = TryCreatePipelineCache(initialData, initialLength, out _pipelineCache); + if (result != Result.Success && initialLength != 0) + { + Console.Error.WriteLine($"[LOADER][WARN] Vulkan pipeline cache rejected ({result}); rebuilding it."); + result = TryCreatePipelineCache(null, 0, out _pipelineCache); + } + } + finally + { + System.Runtime.InteropServices.NativeMemory.Free(initialData); } if (result != Result.Success) @@ -1282,7 +1560,7 @@ internal static unsafe partial class VulkanVideoPresenter return; } - if (initialData.Length != 0 && QueryPipelineCacheSize(_pipelineCache) is { } loadedBytes) + if (initialLength != 0 && QueryPipelineCacheSize(_pipelineCache) is { } loadedBytes) { _pipelineCacheSavedBytes = loadedBytes; } @@ -1307,7 +1585,7 @@ internal static unsafe partial class VulkanVideoPresenter else { Console.Error.WriteLine( - $"[LOADER][INFO] Vulkan pipeline cache ready: path={_pipelineCachePath} initial={initialData.Length} bytes"); + $"[LOADER][INFO] Vulkan pipeline cache ready: path={_pipelineCachePath} initial={initialLength} bytes"); } } @@ -1324,6 +1602,21 @@ internal static unsafe partial class VulkanVideoPresenter return PipelineCacheSignature.Build(properties.VendorID, properties.DeviceID, properties.DriverVersion, uuid); } + private Result TryCreatePipelineCache(byte* initialData, ulong initialLength, out PipelineCache pipelineCache) + { + var createInfo = new PipelineCacheCreateInfo + { + SType = StructureType.PipelineCacheCreateInfo, + InitialDataSize = (nuint)initialLength, + PInitialData = initialLength == 0 ? null : initialData, + }; + return _vk.CreatePipelineCache( + _device, + &createInfo, + null, + out pipelineCache); + } + private Result TryCreatePipelineCache(byte[] initialData, out PipelineCache pipelineCache) { fixed (byte* initialDataPointer = initialData) @@ -1466,73 +1759,46 @@ internal static unsafe partial class VulkanVideoPresenter try { nuint size = 0; - var result = _vk.GetPipelineCacheData( - _device, - cache, - &size, - null); - if (result != Result.Success || size == 0) - { - Console.Error.WriteLine( - $"[LOADER][WARN] Vulkan pipeline cache query failed: result={result} size={size}"); - return false; - } - + var result = _vk.GetPipelineCacheData(_device, cache, &size, null); + if (result != Result.Success || size == 0) return false; if (size > byteLimit) { if ((_resetPipelineCachePaths ??= new HashSet(StringComparer.Ordinal)).Add(path)) { WritePipelineCacheFile(path, []); - Console.Error.WriteLine( - $"[LOADER][INFO] Vulkan pipeline cache outgrew its limit; the next launch starts a fresh cache: path={path} bytes={size} limit={byteLimit}"); + Console.Error.WriteLine($"[LOADER][INFO] Vulkan pipeline cache shard outgrew its limit: path={path} bytes={size} limit={byteLimit}"); } - savedBytes = size; return true; } - - var data = new byte[checked((int)size)]; - fixed (byte* dataPointer = data) + var data = (byte*)System.Runtime.InteropServices.NativeMemory.Alloc(size); + try { - result = _vk.GetPipelineCacheData( - _device, - cache, - &size, - dataPointer); + result = _vk.GetPipelineCacheData(_device, cache, &size, data); + if (result != Result.Success) + { + Console.Error.WriteLine($"[LOADER][WARN] Vulkan pipeline cache export failed: {result}"); + return false; + } + var temporaryPath = path + $".{Environment.ProcessId}.tmp"; + Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(path))!); + using (var file = File.Create(temporaryPath)) + PipelineCacheSignature.WriteTo(file, DriverCacheSignature(), data, size); + File.Move(temporaryPath, path, overwrite: true); + savedBytes = size; + return true; } - - if (result != Result.Success) - { - Console.Error.WriteLine( - $"[LOADER][WARN] Vulkan pipeline cache export failed: {result}"); - return false; - } - - if (size != (nuint)data.Length) - { - Array.Resize(ref data, checked((int)size)); - } - - WritePipelineCacheFile(path, data); - savedBytes = size; - return true; + finally { System.Runtime.InteropServices.NativeMemory.Free(data); } } catch (Exception exception) { - Console.Error.WriteLine( - $"[LOADER][WARN] Vulkan pipeline cache save failed: {exception.Message}"); + Console.Error.WriteLine($"[LOADER][WARN] Vulkan pipeline cache write failed: {exception.Message}"); return false; } } - private void WritePipelineCacheFile(string path, ReadOnlySpan data) { - var directory = Path.GetDirectoryName(path); - if (!string.IsNullOrWhiteSpace(directory)) - { - Directory.CreateDirectory(directory); - } - + Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(path))!); var temporaryPath = path + $".{Environment.ProcessId}.tmp"; File.WriteAllBytes(temporaryPath, PipelineCacheSignature.Wrap(DriverCacheSignature(), data)); File.Move(temporaryPath, path, overwrite: true); diff --git a/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.Device.Teardown.cs b/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.Device.Teardown.cs index cb8bbdb..a7f9647 100644 --- a/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.Device.Teardown.cs +++ b/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.Device.Teardown.cs @@ -34,7 +34,14 @@ internal static unsafe partial class VulkanVideoPresenter DestroyFeedbackSnapshotPool(); DestroyRenderPipelines(); _descriptorHeap.Dispose(); - _imageCache.Dispose(); + if (_imageCache is not null) + { + _imageCache.BindlessImageInvalidator = null; + } + DestroyRuntimeDescriptors(); + _bindlessImageHeap?.Dispose(); + _bindlessImageHeap = null; + _imageCache?.Dispose(); _samplerStore.Dispose(); _bufferCache.AsyncReadback?.Dispose(); _bufferCache.AsyncReadback = null; diff --git a/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.GuestBuffers.cs b/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.GuestBuffers.cs index d8c8e71..dfd9157 100644 --- a/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.GuestBuffers.cs +++ b/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.GuestBuffers.cs @@ -87,9 +87,9 @@ internal static unsafe partial class VulkanVideoPresenter var (start, size) = GlobalBufferRange(guestBuffer); var (buffer, offset) = _bufferCache.ObtainBuffer(start, size, guestBuffer.Writable, isTexelBuffer: formatted); - if (formatted && guestBuffer.Writable) + if (guestBuffer.Writable && !formatted) { - _imageCache.InvalidateMemoryFromGpu(guestBuffer.BaseAddress, guestBuffer.Size); + _imageCache.InvalidateMemoryCopiesFromGpu(guestBuffer.BaseAddress, guestBuffer.Size); } if (offset % _minStorageBufferOffsetAlignment != 0) { diff --git a/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.GuestDraws.cs b/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.GuestDraws.cs index 83a7799..6c09388 100644 --- a/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.GuestDraws.cs +++ b/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.GuestDraws.cs @@ -90,10 +90,15 @@ internal static unsafe partial class VulkanVideoPresenter (4, 4, _) => Format.R32Uint, (4, 5, _) => Format.R32Sint, (4, 7, _) => Format.R32Sfloat, + (5, 1, _) => Format.R16G16SNorm, + (10, 1, _) => Format.R8G8B8A8SNorm, + (12, 1, _) => Format.R16G16B16A16SNorm, (5, 4, _) => Format.R16G16Uint, (5, 5, _) => Format.R16G16Sint, (5, 7, _) => Format.R16G16Sfloat, - (6, 7, _) or (7, 7, _) => Format.B10G11R11UfloatPack32, + // 11_11_10 holds unsigned floats whatever the number type; the image path maps its + // UNORM variant to the same host format (GuestPixelFormats.HostFormats). + (6, 7, _) or (7, 7, _) or (6, 0, _) => Format.B10G11R11UfloatPack32, (9, _, 1) => Format.A2R10G10B10UnormPack32, (9, _, _) => Format.A2B10G10R10UnormPack32, (10, 4, _) => Format.R8G8B8A8Uint, @@ -111,6 +116,13 @@ internal static unsafe partial class VulkanVideoPresenter (13, 7, _) or (14, 7, _) => Format.R32G32B32A32Sfloat, (20, 0, _) => Format.R32Uint, (29, 0, _) or (4, 0, _) => Format.R32Sfloat, + (1, 1, _) => Format.R8SNorm, + (1, 5, _) => Format.R8Sint, + (1, 6 or 9, _) => Format.R8Srgb, + (3, 1, _) => Format.R8G8SNorm, + (3, 4, _) => Format.R8G8Uint, + (3, 5, _) => Format.R8G8Sint, + (3, 6 or 9, _) => Format.R8G8Srgb, (1, 0, _) or (36, 0, _) => Format.R8Unorm, (1, 4, _) or (49, 0, _) => Format.R8Uint, (3, 0, _) => Format.R8G8Unorm, diff --git a/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.GuestSubmissions.cs b/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.GuestSubmissions.cs index 4c56738..531f35f 100644 --- a/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.GuestSubmissions.cs +++ b/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.GuestSubmissions.cs @@ -135,7 +135,7 @@ internal static unsafe partial class VulkanVideoPresenter } while (_pendingGuestSubmissions.TryPeek(out var submission) && - _scheduler.IsTickComplete(submission.Tick)) + (waitForOldest ? _scheduler.IsTickComplete(submission.Tick) : _scheduler.IsTickCompletePolled(submission.Tick))) { _pendingGuestSubmissions.Dequeue(); RetireGuestSubmission(submission); @@ -145,7 +145,7 @@ internal static unsafe partial class VulkanVideoPresenter // Deferred tick work (buffer erases, one-shot uploads, fault parses) runs here. if (_scheduler.Active) { - _scheduler.RunCompletedOperations(); + _scheduler.RunCompletedOperations(polled: !waitForOldest); } _commandStream.NotifyCompletedGpuTick(_scheduler.Timeline.CompletedTick); diff --git a/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.HostMovie.cs b/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.HostMovie.cs index 419959e..e2bfcd5 100644 --- a/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.HostMovie.cs +++ b/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.HostMovie.cs @@ -347,6 +347,7 @@ internal static unsafe partial class VulkanVideoPresenter RowLength = width, DestinationSelect = texture.DstSelect, NeedsUpload = needsUpload, + IsResident = true, IsHostMovie = true, HostMoviePlane = plane, HostMovieFrameSerial = _hostMovieFrameSerial, diff --git a/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.Pipelines.cs b/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.Pipelines.cs index 3b0e354..a3e42ab 100644 --- a/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.Pipelines.cs +++ b/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.Pipelines.cs @@ -44,6 +44,7 @@ internal static unsafe partial class VulkanVideoPresenter public DescriptorSetLayout SetLayout; public DescriptorSetDemand Demand; public bool UsesPushDescriptors; + public bool UsesBindlessImages; public GraphicsPipelineDescription? Description; public Pipeline StripVariant; public Pipeline ListVariant; @@ -52,7 +53,7 @@ internal static unsafe partial class VulkanVideoPresenter public ulong ProfilePixelHash; public ulong ProfileComputeHash; - public bool RectangleList => Description is { StaticParameters.Topology: PrimitiveTopology.PatchList }; + public bool RectangleList => Description is { Tessellation: null, StaticParameters.Topology: PrimitiveTopology.PatchList }; } private readonly Dictionary _shaderModules = new(); @@ -61,8 +62,20 @@ internal static unsafe partial class VulkanVideoPresenter private long _pipelineCreationMilliseconds; private KhrPushDescriptor _pushDescriptorApi = null!; private uint _maxPushDescriptors; + private uint _maxPerStageSampledImages = uint.MaxValue; + private uint _maxPerStageStorageImages = uint.MaxValue; + private uint _maxPerStageStorageBuffers = uint.MaxValue; + private uint _maxPerStageUpdateAfterBindSampledImages = uint.MaxValue; + private uint _maxPerStageUpdateAfterBindStorageImages = uint.MaxValue; + private uint _maxUpdateAfterBindSampledImages = uint.MaxValue; + private uint _maxUpdateAfterBindStorageImages = uint.MaxValue; + private uint _maxUpdateAfterBindDescriptors = uint.MaxValue; + private uint _maxPerStageSamplers = uint.MaxValue; + private uint _maxPerStageUpdateAfterBindSamplers = uint.MaxValue; + private uint _maxUpdateAfterBindSamplers = uint.MaxValue; private SampleCountFlags _noAttachmentSampleCounts; private DescriptorHeap _descriptorHeap = null!; + private BindlessImageHeap? _bindlessImageHeap; uint IShaderPipelineHost.MaxPushDescriptors => _maxPushDescriptors; @@ -73,6 +86,8 @@ internal static unsafe partial class VulkanVideoPresenter // other compute translation maps a guest wave to 32-lane host subgroups, and a 64-lane host // subgroup (AMD's default) left lanes 32..63 inactive: a wave64 8x8x8 group lost rows 4..7. private const uint RdnaSubgroupSize = 32; + + // Pipeline creations at least this slow are logged; cached ones take well under 1 ms. private bool _canRequireComputeSubgroup32; private uint _maxComputeWorkgroupSubgroups; @@ -87,12 +102,26 @@ internal static unsafe partial class VulkanVideoPresenter bool IShaderPipelineHost.GraphicsSubgroupOperationsEnabled => GraphicsSubgroupOperationsEnabled; bool IShaderPipelineHost.SharedInt64AtomicsEnabled => SharedInt64AtomicsEnabled; + bool IShaderPipelineHost.ExactFloat16ConversionsEnabled => ExactFloat16ConversionsEnabled; + + bool IShaderPipelineHost.NonUniformImageIndexingEnabled => NonUniformImageIndexingEnabled; + bool IShaderPipelineHost.RuntimeBufferStridesEnabled => true; + bool IShaderPipelineHost.UsesBindlessImages => BindlessImageHeapEnabled; bool IShaderPipelineHost.ShaderSignedZeroInfNanPreserveFloat32Supported => _supportsShaderSignedZeroInfNanPreserveFloat32; // NVIDIA's compiler rejects the elided-EXEC wave64 compute module with NVVM error 3. bool IShaderPipelineHost.ExecGuardElisionEnabled => _physicalDeviceVendorId != NvidiaVendorId; + + bool IShaderPipelineHost.SupportsDualSourceBlend => _supportsDualSourceBlend; bool IShaderPipelineHost.PerVertexPixelInputsSupported => _supportsPerVertexPixelInputs; bool IShaderPipelineHost.ClipDistanceEnabled => _supportsShaderClipDistance; + private bool _postDepthCoverageEnabled; + bool IShaderPipelineHost.PostDepthCoverageSupported => _postDepthCoverageEnabled; + public bool NativeTwoSampleMixedSupported => _supportsNativeTwoSampleMixed; + + bool IShaderPipelineHost.NativeHalfConversionExact => NativeHalfConversionExact; + + bool IShaderPipelineHost.ZeroOutOfBoundsBufferReads => ZeroOutOfBoundsBufferReads; RenderHostLimits IShaderPipelineHost.Limits => _renderHostLimits; @@ -104,7 +133,7 @@ internal static unsafe partial class VulkanVideoPresenter using var profile = ResourceMaterializationProfile.Measure(ResourceMaterializationProfile.Phase.GuestRead); word = 0; var synchronized = false; - if (IsCleanReadPage(address, sizeof(uint))) + if (IsCleanReadPage(address, sizeof(uint), clean: false)) { if (TryGetAliasPointer(address, sizeof(uint), out var alias)) { @@ -141,7 +170,7 @@ internal static unsafe partial class VulkanVideoPresenter } else { - NoteCleanReadPage(address, sizeof(uint)); + NoteCleanReadPage(address, sizeof(uint), clean: false); } } @@ -165,14 +194,47 @@ internal static unsafe partial class VulkanVideoPresenter { using var profile = ResourceMaterializationProfile.Measure(ResourceMaterializationProfile.Phase.CleanGuestRead); word = 0; + if (!_bufferCache.HasGpuDirtyPages(address, sizeof(uint)) && IsCleanReadPage(address, sizeof(uint), clean: true)) + { + if (TryGetAliasPointer(address, sizeof(uint), out var alias)) + { + word = System.Runtime.CompilerServices.Unsafe.ReadUnaligned(alias); + return true; + } + + Span resident = stackalloc byte[sizeof(uint)]; + if (_guestMemory.TryRead(address, resident)) + { + word = System.Buffers.Binary.BinaryPrimitives.ReadUInt32LittleEndian(resident); + return true; + } + } + + CleanReadVerifications++; if (!_guestMemory.CanRead(address, sizeof(uint))) { return false; } + // A word a previous GPU pass wrote is read once that pass's bytes are back in guest + // memory, as the command processor would see them: an image's contents go through + // its buffer view first, then the buffer comes back to guest memory. + if (_imageCache.HasUnsynchronizedGpuImage(address, sizeof(uint))) + { + foreach (var (imageAddress, imageSize) in _imageCache.UnsynchronizedGpuImageRanges(address, sizeof(uint))) + { + _ = _bufferCache.ObtainBuffer(imageAddress, imageSize, isWritten: false, isTexelBuffer: true); + } + } + + if (_bufferCache.HasGpuDirtyPages(address, sizeof(uint)) || _bufferCache.HasGpuDirtyBytes(address, sizeof(uint))) + { + _ = _bufferCache.TrySynchronizeCpuRead(address, sizeof(uint)); + } + if ((_bufferCache.MayHaveGpuDirtyPages(address, sizeof(uint)) && _bufferCache.HasGpuDirtyPages(address, sizeof(uint))) || _bufferCache.HasGpuDirtyBytes(address, sizeof(uint)) || - _imageCache.HasGpuModifiedImageBytes(address, sizeof(uint))) + _imageCache.HasUnsynchronizedGpuImage(address, sizeof(uint))) { return false; } @@ -183,6 +245,7 @@ internal static unsafe partial class VulkanVideoPresenter return false; } + NoteCleanReadPage(address, sizeof(uint), clean: true); word = System.Buffers.Binary.BinaryPrimitives.ReadUInt32LittleEndian(bytes); return true; } @@ -191,16 +254,28 @@ internal static unsafe partial class VulkanVideoPresenter public bool TryReadResidentGuestBytes(ulong address, Span destination, bool clean) { var size = (ulong)destination.Length; - if (clean || !IsCleanReadPage(address, size)) + if (!_guestMemory.CanRead(address, size)) { + return false; + } + + if (_bufferCache.HasGpuDirtyPages(address, size) || !IsCleanReadPage(address, size, clean)) + { + CleanReadVerifications++; if (!_guestMemory.CanRead(address, size) || _bufferCache.HasGpuDirtyBytes(address, size) || - (clean && (_bufferCache.HasGpuDirtyPages(address, size) || _imageCache.HasGpuModifiedImageBytes(address, size)))) + (clean && _imageCache.HasGpuModifiedImageBytes(address, size))) { return false; } - NoteCleanReadPage(address, size); + // GPU ownership is tracked in bytes, but protection covers whole pages. + // Read disjoint CPU-owned bytes through the backing alias so a protected + // guest view cannot trigger a download of neighboring GPU-owned bytes. + if (clean && _bufferCache.HasGpuDirtyPages(address, size)) + return _guestBacking.TryReadBacking(address, destination); + + NoteCleanReadPage(address, size, clean); } if (TryGetAliasPointer(address, size, out var alias)) @@ -212,6 +287,9 @@ internal static unsafe partial class VulkanVideoPresenter return _guestMemory.TryRead(address, destination); } + bool IRenderHost.TryReadCleanGuestBytes(ulong address, Span destination) => + TryReadResidentGuestBytes(address, destination, clean: true); + private const ulong CleanReadPageBytes = 0x1000; private const int CleanReadPageSlots = 64; private CleanReadPages? _cleanReadPages; @@ -223,8 +301,14 @@ internal static unsafe partial class VulkanVideoPresenter public readonly long[] Versions = new long[CleanReadPageSlots]; public readonly ulong[] Aliases = new ulong[CleanReadPageSlots]; public readonly object?[] Snapshots = new object?[CleanReadPageSlots]; + // A clean read also needs no GPU-dirty page and no GPU-modified image over the page. + public readonly bool[] CleanVerified = new bool[CleanReadPageSlots]; + public readonly long[] ImageVersions = new long[CleanReadPageSlots]; } + // Resident reads that had to check GPU ownership instead of trusting a verified page. + internal long CleanReadVerifications { get; private set; } + private bool TryGetAliasPointer(ulong address, ulong size, out byte* pointer) { pointer = null; @@ -257,34 +341,51 @@ internal static unsafe partial class VulkanVideoPresenter return size != 0 && address + size > address && ((address + size - 1) & ~(CleanReadPageBytes - 1)) == page; } - private bool IsCleanReadPage(ulong address, ulong size) => + private bool IsCleanReadPage(ulong address, ulong size, bool clean) => _cleanReadPages is { } pages && TryGetCleanReadPage(address, size, out var page, out var slot) && pages.Tags[slot] == page + 1 && - pages.Versions[slot] == _bufferCache.GpuModifiedVersion; + pages.Versions[slot] == _bufferCache.GpuModifiedVersion && + (!clean || (pages.CleanVerified[slot] && pages.ImageVersions[slot] == CachedImage.GpuWriteVersion)); - private void NoteCleanReadPage(ulong address, ulong size) + private void NoteCleanReadPage(ulong address, ulong size, bool clean) { if (!TryGetCleanReadPage(address, size, out var page, out var slot)) { return; } + // Read the versions first: a change during the checks then invalidates the entry. var version = _bufferCache.GpuModifiedVersion; + var imageVersion = CachedImage.GpuWriteVersion; if (!_guestMemory.CanRead(page, CleanReadPageBytes) || _bufferCache.HasGpuDirtyBytes(page, CleanReadPageBytes)) { return; } + var cleanVerified = clean && !_bufferCache.HasGpuDirtyPages(page, CleanReadPageBytes) && + !_imageCache.HasGpuModifiedImageBytes(page, CleanReadPageBytes); + var pages = _cleanReadPages ??= new CleanReadPages(); if (pages.Tags[slot] != page + 1) { pages.Tags[slot] = page + 1; pages.Aliases[slot] = 0; pages.Snapshots[slot] = null; + pages.CleanVerified[slot] = false; + } + + if (pages.Versions[slot] != version) + { + pages.CleanVerified[slot] = false; } pages.Versions[slot] = version; + if (cleanVerified) + { + pages.CleanVerified[slot] = true; + pages.ImageVersions[slot] = imageVersion; + } } public ulong CreateShaderModule(IGuestCompiledShader shader, ShaderStage stage, ulong hash, ulong programId) @@ -307,7 +408,7 @@ internal static unsafe partial class VulkanVideoPresenter // The Metal shader compiler can spend seconds on one translated program and the // command stream cannot advance while it does, so a slow creation is reported with // the SPIR-V size that produced it. - private const long SlowPipelineCreationMilliseconds = 250; + private const long SlowPipelineCreationMilliseconds = 500; private int SpirvBytesOf(ulong moduleHandle) => _shaderModuleSpirvBytes.TryGetValue(moduleHandle, out var bytes) ? bytes : -1; @@ -506,6 +607,11 @@ internal static unsafe partial class VulkanVideoPresenter { foreach (var binding in layout.Descriptors) { + if (layout.UsesBindlessImages && ImageDescriptorBinding.ResourceClass(binding.Kind) != ShaderCompiler.Resources.ImageResourceClass.None) + { + continue; + } + bindings.Add(new DescriptorSetLayoutBinding { Binding = BindingLayout.NativeBindingIndex(stage, binding.Kind), @@ -516,6 +622,36 @@ internal static unsafe partial class VulkanVideoPresenter } } + private void EnsureBindlessImageHeap(params ShaderProgramInfo?[] programs) + => EnsureBindlessImageHeapForMode(programs.Any(program => program?.Bindings?.UsesBindlessImages == true)); + + private void EnsureBindlessImageHeapForMode(bool usesBindlessImages) + { + if (_bindlessImageHeap is not null) + { + return; + } + + if (!usesBindlessImages) + { + return; + } + + _bindlessImageHeap = new BindlessImageHeap(_deviceInfo, _scheduler, new BindlessImageHeap.Limits( + _maxPerStageSampledImages, + _maxPerStageStorageImages, + _maxPerStageSamplers, + _maxPerStageUpdateAfterBindSampledImages, + _maxPerStageUpdateAfterBindStorageImages, + _maxPerStageUpdateAfterBindSamplers, + _maxUpdateAfterBindSampledImages, + _maxUpdateAfterBindStorageImages, + _maxUpdateAfterBindSamplers, + _maxUpdateAfterBindDescriptors)); + _imageCache.BindlessImageInvalidator = _bindlessImageHeap.InvalidateViews; + _bindlessImageHeap.SetDefaultSampler(_samplerStore.GetSampler(new SamplerDescriptorWords(stackalloc uint[4]), integerView: false)); + } + // The set is pushed when its descriptors fit the device limit, else it comes from the heap. private DescriptorSetLayout CreateDescriptorSetLayout(List bindings, out bool usesPushDescriptors, out DescriptorSetDemand demand) { @@ -527,6 +663,21 @@ internal static unsafe partial class VulkanVideoPresenter demand = demand.Add(DescriptorSetDemand.Of(binding.DescriptorType, binding.DescriptorCount)); } + // A zero limit is an unreported one. + if (_maxPerStageStorageBuffers != 0 && demand.StorageBuffers > _maxPerStageStorageBuffers) + { + throw SubmissionScheduler.Fatal( + $"The shader stage binds more buffers than the device allows: storage={demand.StorageBuffers}/{_maxPerStageStorageBuffers}."); + } + + if ((_maxPerStageSampledImages != 0 && demand.SampledImages > _maxPerStageSampledImages) || + (_maxPerStageStorageImages != 0 && demand.StorageImages > _maxPerStageStorageImages)) + { + throw SubmissionScheduler.Fatal( + $"The shader stage binds more images than the device allows: sampled={demand.SampledImages}/{_maxPerStageSampledImages} " + + $"storage={demand.StorageImages}/{_maxPerStageStorageImages}."); + } + usesPushDescriptors = descriptorCount <= _maxPushDescriptors; var bindingArray = bindings.ToArray(); fixed (DescriptorSetLayoutBinding* bindingPointer = bindingArray) @@ -543,19 +694,23 @@ internal static unsafe partial class VulkanVideoPresenter } } - private PipelineLayout CreatePipelineLayout(DescriptorSetLayout setLayout, ShaderStageFlags pushStages) + private PipelineLayout CreatePipelineLayout(DescriptorSetLayout setLayout, ShaderStageFlags pushStages, bool usesBindlessImages) { var pushConstants = new PushConstantRange { StageFlags = pushStages, Offset = 0, Size = PushConstantBytes }; + var setLayouts = usesBindlessImages ? new[] { _bindlessImageHeap!.Layout, setLayout } : new[] { setLayout }; var create = new PipelineLayoutCreateInfo { SType = StructureType.PipelineLayoutCreateInfo, - SetLayoutCount = 1, - PSetLayouts = &setLayout, + SetLayoutCount = (uint)setLayouts.Length, PushConstantRangeCount = 1, PPushConstantRanges = &pushConstants, }; - Check(_vk.CreatePipelineLayout(_device, &create, null, out var layout), "vkCreatePipelineLayout"); - return layout; + fixed (DescriptorSetLayout* setLayoutPointer = setLayouts) + { + create.PSetLayouts = setLayoutPointer; + Check(_vk.CreatePipelineLayout(_device, &create, null, out var layout), "vkCreatePipelineLayout"); + return layout; + } } private PipelineHandle RegisterPipeline(RenderPipelineEntry entry) @@ -575,8 +730,22 @@ internal static unsafe partial class VulkanVideoPresenter throw SubmissionScheduler.Fatal($"The draw binds more color attachments than the device supports: count={description.Rendering.ColorCount} max={_maxColorAttachments}."); } + EnsureBindlessImageHeap(description.VertexStage, description.PixelStage); + var usesBindlessImages = description.VertexStage.Bindings!.UsesBindlessImages || + description.PixelStage?.Bindings?.UsesBindlessImages == true; + var bindings = new List(); - CollectLayoutBindings(bindings, description.VertexStage, ShaderStage.Vertex); + CollectLayoutBindings(bindings, description.VertexStage, + description.Tessellation is null ? ShaderStage.Vertex : ShaderStage.TessellationEvaluation); + if (description.Tessellation is not null) + { + var dataBinding = BindingLayout.NativeBindingIndex(ShaderStage.TessellationEvaluation, DescriptorBindingKind.ShaderData); + var index = bindings.FindIndex(binding => binding.Binding == dataBinding); + if (index < 0) throw SubmissionScheduler.Fatal("The tessellation pipeline has no runtime data binding."); + var binding = bindings[index]; + binding.StageFlags |= ShaderStageFlags.TessellationControlBit; + bindings[index] = binding; + } if (description.PixelStage is { } pixelStage) { CollectLayoutBindings(bindings, pixelStage, ShaderStage.Pixel); @@ -587,8 +756,10 @@ internal static unsafe partial class VulkanVideoPresenter { SetLayout = setLayout, Demand = demand, - Layout = CreatePipelineLayout(setLayout, ShaderStageFlags.VertexBit | ShaderStageFlags.FragmentBit), + Layout = CreatePipelineLayout(setLayout, ShaderStageFlags.VertexBit | ShaderStageFlags.FragmentBit | + (description.Tessellation is null ? 0 : ShaderStageFlags.TessellationControlBit | ShaderStageFlags.TessellationEvaluationBit), usesBindlessImages), UsesPushDescriptors = usesPushDescriptors, + UsesBindlessImages = usesBindlessImages, Description = description, ProfileVertexHash = description.VertexStage.Hash, ProfilePixelHash = description.PixelStage?.Hash ?? 0, @@ -742,7 +913,7 @@ internal static unsafe partial class VulkanVideoPresenter var entryPoint = (byte*)SilkMarshal.StringToPtr("main"); try { - var shaderStages = stackalloc PipelineShaderStageCreateInfo[2]; + var shaderStages = stackalloc PipelineShaderStageCreateInfo[4]; var stageCount = 1u; shaderStages[0] = new PipelineShaderStageCreateInfo { @@ -751,6 +922,29 @@ internal static unsafe partial class VulkanVideoPresenter Module = vertexModule, PName = entryPoint, }; + var tessellationState = new PipelineTessellationStateCreateInfo + { + SType = StructureType.PipelineTessellationStateCreateInfo, + PatchControlPoints = description.Tessellation?.InputControlPoints ?? 0, + }; + if (description.Tessellation is { } tessellation) + { + if (topology != PrimitiveTopology.PatchList || tessellation.InputControlPoints is 0 or > 32 || + tessellation.Control.Module == 0 || tessellation.Evaluation.Module == 0) + throw SubmissionScheduler.Fatal("Invalid native tessellation pipeline stages or patch size."); + shaderStages[stageCount++] = new PipelineShaderStageCreateInfo + { + SType = StructureType.PipelineShaderStageCreateInfo, + Stage = ShaderStageFlags.TessellationControlBit, + Module = new ShaderModule(tessellation.Control.Module), PName = entryPoint, + }; + shaderStages[stageCount++] = new PipelineShaderStageCreateInfo + { + SType = StructureType.PipelineShaderStageCreateInfo, + Stage = ShaderStageFlags.TessellationEvaluationBit, + Module = new ShaderModule(tessellation.Evaluation.Module), PName = entryPoint, + }; + } if (pixelModule.Handle != 0) { shaderStages[stageCount++] = new PipelineShaderStageCreateInfo @@ -847,9 +1041,42 @@ internal static unsafe partial class VulkanVideoPresenter FrontFace = parameters.FrontFaceClockwise ? FrontFace.Clockwise : FrontFace.CounterClockwise, LineWidth = 1, }; + var nativeMixed = _supportsNativeTwoSampleMixed && + parameters.Samples == 2 && rendering.DepthSamples == 2 && colorCount > 0 && + rendering.ColorSamples.Take(colorCount).All(count => count == 1); + var coverageReduction = new PipelineCoverageReductionStateCreateInfoNV + { + SType = StructureType.PipelineCoverageReductionStateCreateInfoNV, + CoverageReductionMode = CoverageReductionModeNV.TruncateNV, + }; + var sampleLocationsEnabled = _supportsNativeTwoSampleMixed && parameters.Samples == 2; + var sampleLocations = stackalloc SampleLocationEXT[8]; + for (var pixel = 0; pixel < 4; pixel++) + for (var sample = 0; sample < 2; sample++) + { + var word = rendering.SampleLocationWords[pixel * 4] >> (sample * 8); + var x = (int)(word << 28) >> 28; + var y = (int)(word << 24) >> 28; + sampleLocations[pixel * 2 + sample] = new SampleLocationEXT((x + 8) / 16f, (y + 8) / 16f); + } + var sampleLocationState = new PipelineSampleLocationsStateCreateInfoEXT + { + SType = StructureType.PipelineSampleLocationsStateCreateInfoExt, + PNext = nativeMixed ? &coverageReduction : null, + SampleLocationsEnable = true, + SampleLocationsInfo = new SampleLocationsInfoEXT + { + SType = StructureType.SampleLocationsInfoExt, + SampleLocationsPerPixel = SampleCountFlags.Count2Bit, + SampleLocationGridSize = new Extent2D(2, 2), + SampleLocationsCount = 8, + PSampleLocations = sampleLocations, + }, + }; var multisample = new PipelineMultisampleStateCreateInfo { SType = StructureType.PipelineMultisampleStateCreateInfo, + PNext = sampleLocationsEnabled ? &sampleLocationState : (nativeMixed ? &coverageReduction : null), SampleShadingEnable = parameters.SampleShadingEnable, RasterizationSamples = ImageDescription.VulkanSampleCount(parameters.Samples), MinSampleShading = 1f, @@ -868,10 +1095,10 @@ internal static unsafe partial class VulkanVideoPresenter StencilTestEnable = parameters.StencilTestEnable, Front = ToVkStencilOpState(parameters.StencilFront), Back = ToVkStencilOpState(parameters.StencilBack), - MinDepthBounds = parameters.DepthMinBounds, - MaxDepthBounds = parameters.DepthMaxBounds, + MinDepthBounds = 0f, + MaxDepthBounds = 1f, }; - var dynamicStates = stackalloc DynamicState[13]; + var dynamicStates = stackalloc DynamicState[14]; dynamicStates[0] = DynamicState.Viewport; dynamicStates[1] = DynamicState.Scissor; dynamicStates[2] = DynamicState.LineWidth; @@ -885,6 +1112,11 @@ internal static unsafe partial class VulkanVideoPresenter dynamicStates[10] = DynamicState.StencilWriteMask; dynamicStates[11] = DynamicState.BlendConstants; var dynamicStateCount = 12u; + if (_supportsDepthBounds) + { + dynamicStates[dynamicStateCount++] = DynamicState.DepthBounds; + } + // Last so a pipeline without color attachments can leave it out. if (_colorWriteEnableApi is not null && colorCount != 0) { @@ -905,14 +1137,26 @@ internal static unsafe partial class VulkanVideoPresenter DepthAttachmentFormat = rendering.DepthFormat, StencilAttachmentFormat = rendering.StencilFormat, }; + var colorSampleCounts = stackalloc SampleCountFlags[8]; + for (var index = 0; index < colorCount; index++) + colorSampleCounts[index] = ImageDescription.VulkanSampleCount(rendering.ColorSamples[index]); + var attachmentSampleCounts = new AttachmentSampleCountInfoNV + { + SType = StructureType.AttachmentSampleCountInfoNV, + PNext = &renderingInfo, + ColorAttachmentCount = (uint)colorCount, + PColorAttachmentSamples = colorSampleCounts, + DepthStencilAttachmentSamples = ImageDescription.VulkanSampleCount(rendering.DepthSamples == 0 ? 1 : rendering.DepthSamples), + }; var pipelineInfo = new GraphicsPipelineCreateInfo { SType = StructureType.GraphicsPipelineCreateInfo, - PNext = &renderingInfo, + PNext = nativeMixed ? &attachmentSampleCounts : &renderingInfo, StageCount = stageCount, PStages = shaderStages, PVertexInputState = &vertexInput, PInputAssemblyState = &inputAssembly, + PTessellationState = description.Tessellation is null ? null : &tessellationState, PViewportState = &viewportState, PRasterizationState = &rasterization, PMultisampleState = &multisample, @@ -953,10 +1197,11 @@ internal static unsafe partial class VulkanVideoPresenter public PipelineHandle CreateComputePipeline(ComputePipelineDescription description) { using var profileScope = RenderPhaseProfile.MeasureDetail(RenderPhaseProfile.Phase.PipelineSetup); + EnsureBindlessImageHeap(description.Stage); var bindings = new List(); CollectLayoutBindings(bindings, description.Stage, ShaderStage.Compute); var setLayout = CreateDescriptorSetLayout(bindings, out var usesPushDescriptors, out var demand); - var layout = CreatePipelineLayout(setLayout, ShaderStageFlags.ComputeBit); + var layout = CreatePipelineLayout(setLayout, ShaderStageFlags.ComputeBit, description.Stage.Bindings!.UsesBindlessImages); var computeModule = new ShaderModule(description.Program.Module); if (computeModule.Handle == 0) { @@ -1010,6 +1255,7 @@ internal static unsafe partial class VulkanVideoPresenter SetLayout = setLayout, Demand = demand, UsesPushDescriptors = usesPushDescriptors, + UsesBindlessImages = description.Stage.Bindings!.UsesBindlessImages, ProfileComputeHash = description.Stage.Hash, }; return RegisterPipeline(entry); @@ -1080,6 +1326,7 @@ internal static unsafe partial class VulkanVideoPresenter SetLayout = pending.SetLayout, Demand = pending.Demand, UsesPushDescriptors = pending.UsesPushDescriptors, + UsesBindlessImages = description.Stage.Bindings!.UsesBindlessImages, ProfileComputeHash = pending.Hash, }); return true; @@ -1089,7 +1336,8 @@ internal static unsafe partial class VulkanVideoPresenter var bindings = new List(); CollectLayoutBindings(bindings, description.Stage, ShaderStage.Compute); var setLayout = CreateDescriptorSetLayout(bindings, out var usesPushDescriptors, out var demand); - var layout = CreatePipelineLayout(setLayout, ShaderStageFlags.ComputeBit); + EnsureBindlessImageHeap(description.Stage); + var layout = CreatePipelineLayout(setLayout, ShaderStageFlags.ComputeBit, description.Stage.Bindings!.UsesBindlessImages); var computeModule = new ShaderModule(description.Program.Module); if (computeModule.Handle == 0) { diff --git a/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.RenderLoop.cs b/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.RenderLoop.cs index 307d6f9..12d2bc9 100644 --- a/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.RenderLoop.cs +++ b/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.RenderLoop.cs @@ -456,6 +456,9 @@ internal static unsafe partial class VulkanVideoPresenter VideoOutExports.ReportPresentedFrame(); PerfOverlay.RecordPresent(); RenderPhaseProfile.RecordFrame(); + _bufferCache?.DecayCpuWriteHeat(); + _bufferCache?.CheckCoherence(); + _bufferCache?.WatchFrame(); if (_swapchainReadbackPending) { // Diagnostics read back GPU memory and need this frame done. diff --git a/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.Rendering.cs b/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.Rendering.cs index 799601f..b0e9ca1 100644 --- a/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.Rendering.cs +++ b/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.Rendering.cs @@ -4,6 +4,7 @@ namespace SharpEmu.Libs.VideoOut; using SharpEmu.HLE; +using SharpEmu.HLE.GpuMemory; using SharpEmu.Libs.Agc; using SharpEmu.ShaderCompiler; using SharpEmu.Libs.Gpu; @@ -86,7 +87,7 @@ internal static unsafe partial class VulkanVideoPresenter } finally { - owner._scheduler.QueueCompletionAction(storage.Dispose); + // Storage belongs to its cached attachment and is reused across draws. } } } @@ -198,6 +199,20 @@ internal static unsafe partial class VulkanVideoPresenter public void RunPendingOperations() { + // Start of a draw or dispatch: nothing is bound yet, so the tick can end here. + if (_imageCache.MemoryUnderPressure && !_scheduler.InsideTickCallback) + { + // A slow frame can materialize many draws' textures before the next flip. + // Advance collection here, while no new draw bindings can be invalidated. + _imageCache.RunGarbageCollector(endsFrame: false); + } + if ((_imageCache.ScratchOverBudget || _imageCache.RetirementOverBudget || _bufferCache.RetirementOverBudget) && !_scheduler.InsideTickCallback) + { + _scheduler.Finish(); + if (_imageCache.MemoryUnderPressure) + _imageCache.ReleaseUnusedMemory(); + } + RunPendingCommands(); if (_batchOpen && _preparation is null && _bufferCache?.HotWritePending == true) { @@ -338,6 +353,38 @@ internal static unsafe partial class VulkanVideoPresenter _boundDepthLoadState = depth.LoadState; } + public void ConfigureDepthSampleLocations(in DepthAttachmentState depth, ReadOnlySpan words, ImageLayout layout, ImageAspectFlags writeAspects) + { + var image = _imageCache.GetImage(depth.Image); + if ((image.Backing.Flags & ImageCreateFlags.CreateSampleLocationsCompatibleDepthBitExt) == 0) + throw SubmissionScheduler.Fatal("Custom sample positions require a compatible depth image."); + var locations = new SampleLocationEXT[8]; + for (var pixel = 0; pixel < 4; pixel++) + for (var sample = 0; sample < 2; sample++) + { + var word = words[pixel * 4] >> (sample * 8); + var x = (int)(word << 28) >> 28; + var y = (int)(word << 24) >> 28; + locations[pixel * 2 + sample] = new SampleLocationEXT((x + 8) / 16f, (y + 8) / 16f); + } + if (image.Backing.SampleLocations is null || !locations.AsSpan().SequenceEqual(image.Backing.SampleLocations)) + { + if (!depth.LoadClear && (image.Backing.SampleLocations is not null || + image.Backing.State.Layout != ImageLayout.Undefined || image.Backing.SubresourceStates is not null)) + throw SubmissionScheduler.Fatal("Changing depth sample positions requires a guest depth clear."); + var view = depth.Target.Target.Request.View; + if (view.BaseLevel != 0 || view.LevelCount != image.Backing.MipLevels || + view.BaseLayer != 0 || view.LayerCount != image.Backing.Layers) + throw SubmissionScheduler.Fatal("Changing depth sample positions requires a complete image view."); + EndRendering(); + // The transition describes the positions of the previous depth writes. + TransitionDepthAttachment(in depth, layout, writeAspects); + image.Backing.SampleLocations = locations; + return; + } + TransitionDepthAttachment(in depth, layout, writeAspects); + } + public BufferBinding NullBuffer => new(_bufferCache.GetBuffer(GuestBufferCache.NullBufferId).Handle.Handle, 0); public BufferBinding ObtainBuffer(ulong address, ulong size, bool isWritten) @@ -346,6 +393,9 @@ internal static unsafe partial class VulkanVideoPresenter return new BufferBinding(buffer.Handle.Handle, offset); } + public void PrepareBufferAllocations(ReadOnlySpan ranges) => + _bufferCache.PrepareBufferAllocations(ranges); + // The ring takes the bytes unless it would wrap over a prepared binding; then a host buffer of the draw takes them. public BufferBinding UploadTransient(ReadOnlySpan data, uint alignment) { @@ -369,6 +419,14 @@ internal static unsafe partial class VulkanVideoPresenter return new BufferBinding(buffer.Handle, 0); } + public ulong ObtainBufferDeviceAddress(ulong address, ulong size, bool isWritten) + { + // Read-only indices normally use the transient stream fast path. + // Physical pointers require a cache allocation created with BDA usage. + var (buffer, offset) = _bufferCache.ObtainBuffer(address, size, isWritten, requiresDeviceAddress: true); + return checked(buffer.DeviceAddress + offset); + } + public void BindVertexBuffers(ReadOnlySpan bindings, VertexInputInfo input) { if (bindings.IsEmpty) return; @@ -586,8 +644,13 @@ internal static unsafe partial class VulkanVideoPresenter var view = binding.Request.View; for (var layer = view.BaseLayer; layer < view.BaseLayer + view.LayerCount; layer++) { - var tracked = _imageCache.IsMetadataCleared(metadataAddress, layer, out var metadataValue) && (byte)metadataValue == DccClearToZero; + // Every fixed clear code carries its own colour; only the register code (0x20) needs + // the CB clear word, which is known when the surface binds as a target. Silent Hill + // fast-clears UE's separate-translucency target to (0,0,0,1) and only samples it: + // skipping that clear left stale zeros that blacked out the whole title-menu scene. var clearValue = default(ClearColorValue); + var tracked = _imageCache.IsMetadataCleared(metadataAddress, layer, out var metadataValue) && + TryDecodeDccClear((byte)metadataValue, false, (byte)metadataValue == DccClearToZero || fixedClearSupported, default, out clearValue); ulong guestSlice = 0; if (!tracked && (sliceSize == 0 || !_imageCache.TryReadGuestDccClear(metadataAddress, sliceSize, layer, out guestSlice, out var code) || @@ -661,6 +724,11 @@ internal static unsafe partial class VulkanVideoPresenter _vk.CmdSetDepthWriteEnable(command, state.DepthWriteEnabled); _vk.CmdSetDepthCompareOp(command, state.DepthCompare); _vk.CmdSetDepthBiasEnable(command, state.DepthBiasEnabled); + if (_supportsDepthBounds) + { + _vk.CmdSetDepthBounds(command, state.DepthMinBounds, state.DepthMaxBounds); + } + if (state.DepthBiasEnabled) { _vk.CmdSetDepthBias(command, state.DepthBiasConstantFactor, _supportsDepthBiasClamp ? state.DepthBiasClamp : 0f, state.DepthBiasSlopeFactor); @@ -909,6 +977,17 @@ internal static unsafe partial class VulkanVideoPresenter _nextDrawWritesMemory = true; } + public void PrepareGraphicsPipeline(in PipelineHandle pipeline) + { + if (!_supportsNativeTwoSampleMixed || _supportsVariableSampleLocations || !_renderingActive) + return; + var next = RequirePipelineEntry(in pipeline).Description; + var previous = _boundGraphicsPipeline?.Description; + if (next is not null && previous is not null && next.StaticParameters.Samples == 2 && + previous.StaticParameters.Samples == 2 && + !next.Rendering.SampleLocationWords.AsSpan().SequenceEqual(previous.Rendering.SampleLocationWords)) + EndRendering(); + } public void BindPipeline(PipelineBindPoint bindPoint, in PipelineHandle pipeline) { var entry = RequirePipelineEntry(in pipeline); @@ -1042,6 +1121,31 @@ internal static unsafe partial class VulkanVideoPresenter CountDraw(); } + void IRenderHost.DrawIndirect(BufferBinding arguments) + { + using var profileScope = RenderPhaseProfile.MeasureDetail(RenderPhaseProfile.Phase.DrawRecording); + var command = BeginBatchedGuestCommands(); + if (_boundGraphicsPipeline is { RectangleList: true } entry) + { + // The count is unknown here, so neither the single-rectangle strip nor a + // count check for native rectangles applies. + if (_supportsFillRectangle) + { + BindNativeRectangleList(entry, command); + } + else + { + BindRectangleListVariant(entry, strip: false, command); + } + } + + _gpuCommandProfile?.WriteMarker(command, VulkanCommandProfile.IntervalKind.Preparation); + _vk.CmdDrawIndirect(command, new VkBuffer(arguments.Handle), arguments.Offset, 1, 16); + _gpuCommandProfile?.WriteMarker(command, VulkanCommandProfile.IntervalKind.Draw, + _boundGraphicsPipeline?.Id ?? 0, 0, 0); + CountDraw(); + } + public void Dispatch(uint groupsX, uint groupsY, uint groupsZ) { using var profileScope = RenderPhaseProfile.MeasureDetail(RenderPhaseProfile.Phase.DrawRecording); @@ -1091,8 +1195,7 @@ internal static unsafe partial class VulkanVideoPresenter public void ShaderWriteBarrier(PipelineStageFlags sourceStages) => RecordMemoryBarrier( sourceStages, - PipelineStageFlags.ComputeShaderBit | PipelineStageFlags.VertexInputBit | PipelineStageFlags.VertexShaderBit | - PipelineStageFlags.FragmentShaderBit | PipelineStageFlags.TransferBit | PipelineStageFlags.ColorAttachmentOutputBit, + PipelineStageFlags.ComputeShaderBit | PipelineStageFlags.AllGraphicsBit | PipelineStageFlags.TransferBit, AccessFlags.ShaderWriteBit, AccessFlags.ShaderReadBit | AccessFlags.ShaderWriteBit | AccessFlags.VertexAttributeReadBit | AccessFlags.IndexReadBit | AccessFlags.UniformReadBit | AccessFlags.TransferReadBit | AccessFlags.TransferWriteBit | diff --git a/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.RuntimeDescriptors.cs b/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.RuntimeDescriptors.cs new file mode 100644 index 0000000..17b5e51 --- /dev/null +++ b/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.RuntimeDescriptors.cs @@ -0,0 +1,358 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +namespace SharpEmu.Libs.VideoOut; + +using System.Collections.Concurrent; +using System.Runtime.InteropServices; +using SharpEmu.Libs.Gpu.Buffers; +using SharpEmu.Libs.Gpu.Images; +using SharpEmu.Libs.Gpu.Pipelines; +using SharpEmu.Libs.Gpu.Rendering; +using SharpEmu.Libs.Gpu.Scheduling; +using SharpEmu.ShaderCompiler.Resources; +using Silk.NET.Vulkan; + +// This partial resolves the descriptors shaders read at run time (MemoryAccessInfo.RuntimeDescriptor). +// Such a shader looks its image and sampler words up in a table this host builds; a missed +// lookup lands in the miss buffer, which is read when the tick completes. The next draw that +// uses runtime descriptors creates what was missed and rebuilds the table. +// +// Every image registered by this shader goes through the normal texture path on its draws: it is found +// again in the image cache, synchronized with guest memory and transitioned with the draw's +// own textures, so its heap slot always names a current view. +internal static unsafe partial class VulkanVideoPresenter +{ + private sealed partial class Presenter + { + // Up to nine key dwords (view class + image words, or the sampler words) with value equality. + private readonly record struct RuntimeDescriptorKey(ulong Words01, ulong Words23, ulong Words45, ulong Words67, uint Word8, int Count) + { + public static RuntimeDescriptorKey From(ReadOnlySpan words) + { + Span padded = stackalloc uint[9]; + padded.Clear(); + words.CopyTo(padded); + static ulong Pair(Span values, int index) => values[index] | ((ulong)values[index + 1] << 32); + return new RuntimeDescriptorKey(Pair(padded, 0), Pair(padded, 2), Pair(padded, 4), Pair(padded, 6), padded[8], words.Length); + } + } + + private sealed class RuntimeImageEntry(uint[] key) + { + // The view class followed by the eight image words, as the shader hashes them. + public uint[] Key { get; } = key; + public ImageDimension Dimension => RuntimeDescriptorTable.ViewDimension(Key[0]); + public ImageNumericClass NumericClass => RuntimeDescriptorTable.ViewNumericClass(Key[0]); + public uint Slot { get; set; } + // The texture its words resolved to, kept while the cache keeps its image. + public TextureResource? Resolved { get; set; } + } + + private sealed class RuntimeSamplerEntry(uint[] key, uint slot) + { + public uint[] Key { get; } = key; + public uint Slot { get; } = slot; + } + + private sealed class RuntimeDescriptorSet + { + public readonly Dictionary Images = new(); + public readonly Dictionary Samplers = new(); + public readonly ConcurrentQueue<(bool Image, uint[] Key)> Misses = new(); + public readonly TickOwnedResources MissBuffers = new(); + public GpuBuffer? Table; + public bool TableDirty = true; + public ulong MissScanTick = ulong.MaxValue; + } + + // Feedback is owned by the shader that generated it, not by the process. + // In particular, a new scene must not acquire every older shader's textures. + private readonly Dictionary<(ShaderStageKind Stage, ulong Hash), RuntimeDescriptorSet> _runtimeDescriptorSets = new(); + + private RuntimeDescriptorSet RuntimeDescriptorsFor(ShaderProgramInfo program) + { + var key = (program.Stage, program.Hash); + if (!_runtimeDescriptorSets.TryGetValue(key, out var descriptors)) + _runtimeDescriptorSets.Add(key, descriptors = new RuntimeDescriptorSet()); + return descriptors; + } + + // The texture path of a registered image: the view class is part of the key because + // Vulkan image loads need a view whose numeric type matches the SPIR-V image type. + private static ImageResource RuntimeImageResource(RuntimeImageEntry entry) => new() + { + ResourceClass = ShaderCompiler.Resources.ImageResourceClass.Sampled, + NumericClass = entry.NumericClass, + Dimension = entry.Dimension, + }; + + // Registers hold whatever a lane computed, so a runtime descriptor can describe no valid + // view (the hardware reads the null texture then) or name memory that is not mapped, or no + // longer is. An image there would have no guest bytes to track. + private bool IsBackedRuntimeTexture(RuntimeImageEntry entry) + { + if (!ImageRequestBuilders.TryTexture(entry.Key.AsSpan(1), ShapeOf(RuntimeImageResource(entry)), out var resolution)) + { + return false; + } + + var data = resolution.Request.Description.Data; + return ImageDescription.IsEmptyRange(data) || _guestMemory.CanRead(data.Address, data.Size); + } + + // Called while a stage's textures are resolved: registers what earlier draws missed and + // binds this shader's registered images with the draw. + private void PrepareRuntimeDescriptors(PreparedStageBindings prepared) + { + var descriptors = RuntimeDescriptorsFor(prepared.Program); + RegisterRuntimeDescriptorMisses(descriptors); + List? freed = null; + foreach (var (key, entry) in descriptors.Images) + { + if (!TryBindRuntimeImage(entry, prepared.Program, out var binding)) + { + (freed ??= []).Add(key); + continue; + } + + prepared.RuntimeImages.Add(binding); + prepared.RuntimeEntries.Add(entry); + } + + if (freed is not null) + { + foreach (var key in freed) descriptors.Images.Remove(key); + descriptors.TableDirty = true; + } + } + + // An entry keeps the texture its words resolved to while the cache keeps that image; each + // draw binds its own copy. Only a new entry, or one whose image the cache retired, is + // resolved again - and a retired one whose memory the title freed leaves the table, so a + // shader that still reads the descriptor misses it again. + private bool TryBindRuntimeImage(RuntimeImageEntry entry, ShaderProgramInfo program, out TextureResource binding) + { + CachedImage? stale = null; + if (entry.Resolved is { } resolved && !IsStaleImage(resolved.ImageIdentifier, out stale)) + { + BindImage(resolved.ImageIdentifier, storage: false); + binding = CopyRuntimeBinding(resolved); + return true; + } + + if (stale is not null) + { + stale.Binding = default; + } + + entry.Resolved = null; + if (!IsBackedRuntimeTexture(entry)) + { + binding = null!; + return false; + } + + entry.Resolved = ResolveImageBinding(RuntimeImageResource(entry), entry.Key[1..], program, -1); + binding = CopyRuntimeBinding(entry.Resolved); + return true; + } + + // The resolved part of a binding; the acquisition fields belong to one draw. + private static TextureResource CopyRuntimeBinding(TextureResource resolved) => new() + { + Address = resolved.Address, + ImageIdentifier = resolved.ImageIdentifier, + Request = resolved.Request, + IsStorage = resolved.IsStorage, + IsResident = resolved.IsResident, + DestinationSelect = resolved.DestinationSelect, + Width = resolved.Width, + Height = resolved.Height, + }; + + // Called with the stage's other images: acquires each registered view, refreshes its heap + // slot and binds the current table and the miss buffer. + private void BindRuntimeDescriptors(PreparedStageBindings prepared) + { + var descriptors = RuntimeDescriptorsFor(prepared.Program); + var heap = _bindlessImageHeap ?? throw SubmissionScheduler.Fatal("Runtime image descriptors need the bindless image heap."); + for (var index = 0; index < prepared.RuntimeImages.Count; index++) + { + var entry = prepared.RuntimeEntries[index]; + var binding = prepared.RuntimeImages[index]; + var resource = RuntimeImageResource(entry); + if (IsStaleImage(binding.ImageIdentifier, out var stale)) + { + if (stale is not null) + { + stale.Binding = default; + } + + entry.Resolved = ResolveImageBinding(resource, entry.Key[1..], prepared.Program, -1); + prepared.RuntimeImages[index] = binding = CopyRuntimeBinding(entry.Resolved); + } + + // Acquisition refreshes CPU writes and transitions the image for this draw. + var image = _imageCache.GetImage(binding.ImageIdentifier); + binding.View = _imageCache.AcquireTextureView(binding.ImageIdentifier, binding.Request); + binding.MipViews = []; + image.Uses.Texture = true; + binding.CachedImage = image; + binding.Image = image.Backing.Handle; + var kind = ImageDescriptorBinding.ForImage(resource) ?? + throw SubmissionScheduler.Fatal($"A runtime image has no binding class: dimension={entry.Dimension}."); + var slot = heap.GetOrCreateSlot(kind, entry.Key.AsSpan(1), binding.View, binding.Layout); + if (slot != entry.Slot) + { + entry.Slot = slot; + descriptors.TableDirty = true; + } + } + + var table = RuntimeDescriptorTableBuffer(descriptors); + prepared.Descriptors.RuntimeTable = new BufferView(table.Handle, 0, table.Size); + } + + private void CommitRuntimeDescriptorBuffers(PreparedStageBindings prepared) + { + // Image allocation can complete an earlier submission while this + // draw is being prepared. Bind feedback storage to the tick that + // will actually execute the draw, after all image allocations. + var descriptors = RuntimeDescriptorsFor(prepared.Program); + var table = RuntimeDescriptorTableBuffer(descriptors); + var misses = RuntimeDescriptorMissBuffer(descriptors); + prepared.Descriptors.RuntimeTable = new BufferView(table.Handle, 0, table.Size); + prepared.Descriptors.RuntimeMisses = new BufferView(misses.Handle, 0, misses.Size); + ScanRuntimeDescriptorMissesAfterTick(descriptors); + } + + private void RegisterRuntimeDescriptorMisses(RuntimeDescriptorSet descriptors) + { + while (descriptors.Misses.TryDequeue(out var miss)) + { + var key = RuntimeDescriptorKey.From(miss.Key); + if (miss.Image) + { + // A descriptor that describes no host texture (a null base or an invalid format) + // stays out of the table, so the shader keeps reading the null descriptor for + // it, as the hardware does. + if (!descriptors.Images.ContainsKey(key) && RuntimeDescriptorTable.SupportsRuntimeView(RuntimeDescriptorTable.ViewDimension(miss.Key[0])) && + ImageRequestBuilders.DescribesHostTexture(miss.Key.AsSpan(1)) && new RuntimeImageEntry(miss.Key) is var entry && + IsBackedRuntimeTexture(entry)) + { + descriptors.Images.Add(key, entry); + descriptors.TableDirty = true; + } + + continue; + } + + if (!descriptors.Samplers.ContainsKey(key)) + { + var sampler = _samplerStore.GetSampler(new SamplerDescriptorWords(miss.Key), integerView: false); + var slot = _bindlessImageHeap!.GetOrCreateSamplerSlot(sampler); + descriptors.Samplers.Add(key, new RuntimeSamplerEntry(miss.Key, slot)); + descriptors.TableDirty = true; + } + } + } + + // The table buffer of the current entries. A change builds a new buffer: draws already + // recorded keep reading the one they were bound with until their tick completes. + private GpuBuffer RuntimeDescriptorTableBuffer(RuntimeDescriptorSet descriptors) + { + var current = descriptors.Table; + if (current is not null && !descriptors.TableDirty) + { + return current; + } + + var images = descriptors.Images.Values.Where(entry => entry.Slot != 0).Select(entry => (entry.Key, entry.Slot)).ToList(); + var samplers = descriptors.Samplers.Values.Select(entry => (entry.Key, entry.Slot)).ToList(); + var words = RuntimeDescriptorTable.Build(images, samplers); + var buffer = new GpuBuffer(_deviceInfo, _scheduler, GpuBufferUsage.Upload, 0, BufferUsageFlags.StorageBufferBit, (ulong)words.Length * sizeof(uint)); + buffer.Write(0, MemoryMarshal.AsBytes(words.AsSpan())); + if (!buffer.IsCoherent) + { + buffer.Flush(0, buffer.Size); + } + + if (current is not null) + { + _scheduler.QueueCompletionAction(current.Dispose); + } + + descriptors.Table = buffer; + descriptors.TableDirty = false; + return buffer; + } + + private GpuBuffer RuntimeDescriptorMissBuffer(RuntimeDescriptorSet descriptors) + { + return descriptors.MissBuffers.Acquire(_scheduler.CurrentTick, CreateRuntimeDescriptorMissBuffer); + } + + private GpuBuffer CreateRuntimeDescriptorMissBuffer() + { + var buffer = new GpuBuffer(_deviceInfo, _scheduler, GpuBufferUsage.Download, 0, BufferUsageFlags.StorageBufferBit, + RuntimeDescriptorTable.MissBufferDwords * sizeof(uint)); + buffer.Mapped.Clear(); + if (!buffer.IsCoherent) + { + buffer.Flush(0, buffer.Size); + } + + return buffer; + } + + // Once per tick that binds runtime descriptors: the misses its draws recorded are read + // when it completes. Each tick owns separate storage until that read finishes. + private void ScanRuntimeDescriptorMissesAfterTick(RuntimeDescriptorSet descriptors) + { + var tick = _scheduler.CurrentTick; + if (descriptors.MissScanTick == tick) + { + return; + } + + descriptors.MissScanTick = tick; + _scheduler.QueueCompletionAction(() => + descriptors.MissBuffers.Complete(tick, buffer => ReadRuntimeDescriptorMisses(descriptors, buffer))); + } + + private void ReadRuntimeDescriptorMisses(RuntimeDescriptorSet descriptors, GpuBuffer buffer) + { + if (!buffer.IsCoherent) + { + buffer.Invalidate(0, buffer.Size); + } + + var words = MemoryMarshal.Cast(buffer.Mapped); + var imageCount = Math.Min(words[(int)RuntimeDescriptorTable.ImageMissCountDword], RuntimeDescriptorTable.ImageMissCapacity); + var samplerCount = Math.Min(words[(int)RuntimeDescriptorTable.SamplerMissCountDword], RuntimeDescriptorTable.SamplerMissCapacity); + for (uint record = 0; record < imageCount; record++) + { + var start = (int)(RuntimeDescriptorTable.ImageMissOffset + record * RuntimeDescriptorTable.ImageMissRecordDwords); + descriptors.Misses.Enqueue((true, words.Slice(start, (int)RuntimeDescriptorTable.ImageMissRecordDwords).ToArray())); + } + + for (uint record = 0; record < samplerCount; record++) + { + var start = (int)(RuntimeDescriptorTable.SamplerMissOffset + record * RuntimeDescriptorTable.SamplerMissRecordDwords); + descriptors.Misses.Enqueue((false, words.Slice(start, (int)RuntimeDescriptorTable.SamplerMissRecordDwords).ToArray())); + } + + } + + private void DestroyRuntimeDescriptors() + { + foreach (var descriptors in _runtimeDescriptorSets.Values) + { + descriptors.Table?.Dispose(); + descriptors.MissBuffers.Dispose(); + } + _runtimeDescriptorSets.Clear(); + } + } +} diff --git a/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.Scheduler.cs b/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.Scheduler.cs index dbdff71..485ab01 100644 --- a/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.Scheduler.cs +++ b/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.Scheduler.cs @@ -101,7 +101,11 @@ internal static unsafe partial class VulkanVideoPresenter private void CreateImageCache() { var (memory, _, backing) = RequireGuestMemory("image store"); - _imageCache = new GuestImageCache(_deviceInfo, _scheduler, memory.Pages, _bufferCache, backing, readbackLinearImages: false); + // The guest reads linear GPU-written images back on the CPU (UE's exposure readback + // feeds View.PreExposure), so they are published to guest memory after their tick. + // SHARPEMU_READBACK_LINEAR_IMAGES=0 keeps them GPU-only. + _imageCache = new GuestImageCache(_deviceInfo, _scheduler, memory.Pages, _bufferCache, backing, + readbackLinearImages: Environment.GetEnvironmentVariable("SHARPEMU_READBACK_LINEAR_IMAGES") != "0"); _bufferCache.ImageCache = _imageCache; _samplerStore = new SamplerStore(_deviceInfo); } @@ -120,7 +124,10 @@ internal static unsafe partial class VulkanVideoPresenter // Drain accepted submissions before closing the relay. // Cancel blocked submissions if a full retry cycle makes no progress. _commandStream.StopAccepting(); - var outcome = _commandStream.DrainForShutdown(cancelBlockedOnNoProgress: true); + // The runtime may tear down guest memory while this drains; stop executing then. + var outcome = _commandStream.DrainForShutdown( + cancelBlockedOnNoProgress: true, + abandon: static () => GuestGpuMemoryHook.Current is null); FlushBatchedGuestCommands(); Console.Error.WriteLine( $"[LOADER][PERF] command_stream submissions={_commandStream.SubmissionsStarted} " + diff --git a/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.ShaderPrewarm.cs b/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.ShaderPrewarm.cs index d063eae..c9e8a8a 100644 --- a/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.ShaderPrewarm.cs +++ b/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.ShaderPrewarm.cs @@ -93,8 +93,9 @@ internal static unsafe partial class VulkanVideoPresenter var workerCount = Math.Clamp(Environment.ProcessorCount / 4, 1, 4); var remaining = workerCount; var host = (IShaderPipelineHost)this; - var sharedInt64Atomics = host.SharedInt64AtomicsEnabled; - var execGuardElision = host.ExecGuardElisionEnabled; + // The workers create layouts that include the global set. Initialize + // it once on the device setup thread, before any worker can use it. + EnsureBindlessImageHeapForMode(host.UsesBindlessImages); Volatile.Write(ref _shaderPrewarmTotal, pending.Count); Volatile.Write(ref _shaderPrewarmProgress, 0); Console.Error.WriteLine( @@ -107,7 +108,7 @@ internal static unsafe partial class VulkanVideoPresenter { while (!_shaderPrewarmStopping && work.TryDequeue(out var item)) { - PrewarmComputePipeline(item.Record, item.Code, compiler, sharedInt64Atomics, execGuardElision); + PrewarmComputePipeline(item.Record, item.Code, compiler, host); _shaderPrewarmCompleted.Enqueue(ShaderPrewarmList.Identity(item.Record)); if (Interlocked.Increment(ref _shaderPrewarmProgress) % PrewarmProgressBatch == 0) { @@ -158,12 +159,10 @@ internal static unsafe partial class VulkanVideoPresenter ComputePrewarmRecord record, ShaderCodeCapture code, IGuestGpuBackend compiler, - bool sharedInt64Atomics, - bool execGuardElision) + IShaderPipelineHost host) { if (!ShaderProgramCache.TryCompilePrewarm( - record, code, compiler, sharedInt64Atomics, execGuardElision, - _supportsShaderSignedZeroInfNanPreserveFloat32, out var compiled, out var layout, out var error)) + record, code, compiler, host, out var compiled, out var layout, out var error)) { NoteShaderPrewarmFailure(record, error); return; @@ -179,7 +178,7 @@ internal static unsafe partial class VulkanVideoPresenter try { setLayout = CreateDescriptorSetLayout(bindings, out _, out _); - pipelineLayout = CreatePipelineLayout(setLayout, ShaderStageFlags.ComputeBit); + pipelineLayout = CreatePipelineLayout(setLayout, ShaderStageFlags.ComputeBit, layout!.UsesBindlessImages); module = CreateShaderModule(payload); var pipeline = CompileComputePipeline(_vk, _device, _pipelineCache, module, pipelineLayout); _vk.DestroyPipeline(_device, pipeline, null); diff --git a/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.cs b/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.cs index c9a8d52..1291598 100644 --- a/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.cs +++ b/src/SharpEmu.Libs/VideoOut/VulkanVideoPresenter.cs @@ -82,6 +82,18 @@ internal static unsafe partial class VulkanVideoPresenter private static int _nativeSubgroupSize; private static int _nativeSubgroupShaderStages; private static int _sharedInt64AtomicsSupported; + private static int _nonUniformImageIndexingSupported; + private static int _bindlessImageHeapSupported; + + internal static bool NonUniformImageIndexingEnabled => Volatile.Read(ref _nonUniformImageIndexingSupported) != 0; + + internal static bool BindlessImageHeapEnabled => Volatile.Read(ref _bindlessImageHeapSupported) != 0; + + private static void SetNonUniformImageIndexingCapability(bool supported) => + Volatile.Write(ref _nonUniformImageIndexingSupported, supported ? 1 : 0); + + private static void SetBindlessImageHeapCapability(bool supported) => + Volatile.Write(ref _bindlessImageHeapSupported, supported ? 1 : 0); // True when the device supports shaderSharedInt64Atomics, so LDS 64-bit // atomics can be emitted as real 64-bit atomics. @@ -91,12 +103,37 @@ internal static unsafe partial class VulkanVideoPresenter private static void SetSharedInt64AtomicsCapability(bool supported) => Volatile.Write(ref _sharedInt64AtomicsSupported, supported ? 1 : 0); + private static int _exactFloat16ConversionsSupported; + + // True when the device was created with shaderFloat16 and f16 float controls. + internal static bool ExactFloat16ConversionsEnabled => + Volatile.Read(ref _exactFloat16ConversionsSupported) != 0; + + private static void SetExactFloat16ConversionsCapability(bool supported) => + Volatile.Write(ref _exactFloat16ConversionsSupported, supported ? 1 : 0); + internal static bool GraphicsSubgroupOperationsEnabled => VulkanGraphicsSubgroupPolicy.ShouldUseNativeGraphicsSubgroups( unchecked((uint)Volatile.Read(ref _nativeSubgroupSize)), (ShaderStageFlags)Volatile.Read(ref _nativeSubgroupShaderStages), Environment.GetEnvironmentVariable("SHARPEMU_GRAPHICS_SUBGROUPS")); + private static int _nativeHalfConversionExact; + private static int _nativeHalfConversionProbed; + private static int _zeroOutOfBoundsReads; + private static int _zeroOutOfBoundsReadsProbed; + private static bool _robustBufferAccess2Enabled; + + // Set once per process by the device-setup probe: GLSL UnpackHalf2x16 / PackHalf2x16 produced + // exactly what the translator's integer f16 conversion produces, for every test vector. False + // until then, so a device that is never probed keeps the exact emulation. + internal static bool NativeHalfConversionExact => Volatile.Read(ref _nativeHalfConversionExact) != 0; + + // Set once per process by the device-setup probe: a storage-buffer read past the end of its + // descriptor range returned zero on this device, so the translator's own range test, address + // clamp and zero select on every guest buffer word are redundant. False until then. + internal static bool ZeroOutOfBoundsBufferReads => Volatile.Read(ref _zeroOutOfBoundsReads) != 0; + private static void SetNativeSubgroupCapabilities(uint subgroupSize, ShaderStageFlags supportedStages) { Volatile.Write(ref _nativeSubgroupShaderStages, (int)supportedStages); diff --git a/src/SharpEmu.ShaderCompiler.Metal/Gen5MslTranslator.Alu.cs b/src/SharpEmu.ShaderCompiler.Metal/Gen5MslTranslator.Alu.cs index 43a6f83..602746c 100644 --- a/src/SharpEmu.ShaderCompiler.Metal/Gen5MslTranslator.Alu.cs +++ b/src/SharpEmu.ShaderCompiler.Metal/Gen5MslTranslator.Alu.cs @@ -352,8 +352,10 @@ public static partial class Gen5MslTranslator "VLshlOrU32" => $"((({RawSource(instruction, 0)}) << (({RawSource(instruction, 1)}) & 31u)) | ({RawSource(instruction, 2)}))", "VLshlB32" => $"(({RawSource(instruction, 0)}) << (({RawSource(instruction, 1)}) & 31u))", + "VLshlrevB16" => EmitInteger16ShiftReverse(instruction, destination, left: true), "VLshlrevB32" => $"(({RawSource(instruction, 1)}) << (({RawSource(instruction, 0)}) & 31u))", "VLshrB32" => $"(({RawSource(instruction, 0)}) >> (({RawSource(instruction, 1)}) & 31u))", + "VLshrrevB16" => EmitInteger16ShiftReverse(instruction, destination, left: false), "VLshrrevB32" => $"(({RawSource(instruction, 1)}) >> (({RawSource(instruction, 0)}) & 31u))", "VLshrrevB64" => EmitLshrrevB64(instruction, destination), "VAshrI32" => @@ -1261,6 +1263,9 @@ public static partial class Gen5MslTranslator StoreScalar(destination, result); return true; } + case "SFlbitI32B32": + StoreScalar(destination, $"{left} == 0u ? 0xFFFFFFFFu : (uint)clz({left})"); + return true; case "SBitset0B32": StoreScalar( destination, @@ -1714,6 +1719,31 @@ public static partial class Gen5MslTranslator /// Raw 32-bit source with DPP/DPP8 lane remapping on src0 and SDWA /// byte/word selection + integer modifiers, mirroring GetRawSource. /// + private string EmitInteger16ShiftReverse( + Gen5ShaderInstruction instruction, + uint destination, + bool left) + { + var control = instruction.Control as Gen5Vop3Control; + var shift = RawSource(instruction, 0, applySdwaIntegerModifiers: false); + var value = RawSource(instruction, 1, applySdwaIntegerModifiers: false); + if ((control?.OperandSelect & 0x1) != 0) + { + shift = $"(({shift}) >> 16u)"; + } + + if ((control?.OperandSelect & 0x2) != 0) + { + value = $"(({value}) >> 16u)"; + } + + var operation = left ? "<<" : ">>"; + var low16 = $"(((({value}) & 0xFFFFu) {operation} (({shift}) & 0xFu)) & 0xFFFFu)"; + return (control?.OperandSelect & 0x8) != 0 + ? $"((v[{destination}] & 0x0000FFFFu) | (({low16}) << 16u))" + : $"((v[{destination}] & 0xFFFF0000u) | ({low16}))"; + } + private string RawSource( Gen5ShaderInstruction instruction, int sourceIndex, diff --git a/src/SharpEmu.ShaderCompiler.Metal/Gen5MslTranslator.Pixel.cs b/src/SharpEmu.ShaderCompiler.Metal/Gen5MslTranslator.Pixel.cs index 279f2b4..5ef40f1 100644 --- a/src/SharpEmu.ShaderCompiler.Metal/Gen5MslTranslator.Pixel.cs +++ b/src/SharpEmu.ShaderCompiler.Metal/Gen5MslTranslator.Pixel.cs @@ -562,13 +562,16 @@ public static partial class Gen5MslTranslator if (export.Compressed) { var packed = $"v[{instruction.Sources[component >> 1].Value}]"; - var half = $"(float)as_type({packed})[{component & 1}]"; - values[component] = binding.Value.Kind switch + var bits = $"(({packed} >> {(component & 1) * 16}) & 0xffffu)"; + var decoded = binding.Value.ExportFormat switch { - Gen5PixelOutputKind.Uint => $"(uint)({half})", - Gen5PixelOutputKind.Sint => $"(int)({half})", - _ => half, + Gen5PixelExportFormat.Unorm16 => $"(float({bits}) / 65535.0f)", + Gen5PixelExportFormat.Snorm16 => $"max(float(as_type(ushort({bits}))) / 32767.0f, -1.0f)", + Gen5PixelExportFormat.Uint16 => bits, + Gen5PixelExportFormat.Sint16 => $"int(as_type(ushort({bits})))", + _ => $"(float)as_type({packed})[{component & 1}]", }; + values[component] = $"{componentType}({decoded})"; continue; } diff --git a/src/SharpEmu.ShaderCompiler.Metal/Gen5MslTranslator.Resources.cs b/src/SharpEmu.ShaderCompiler.Metal/Gen5MslTranslator.Resources.cs index a36f811..86b4c02 100644 --- a/src/SharpEmu.ShaderCompiler.Metal/Gen5MslTranslator.Resources.cs +++ b/src/SharpEmu.ShaderCompiler.Metal/Gen5MslTranslator.Resources.cs @@ -19,6 +19,11 @@ public static partial class Gen5MslTranslator public static bool TryCompileProgram(ShaderCompileRequest request, out Gen5MslShader shader, out string error) { shader = default!; + if (request.Stage == ShaderStage.TessellationEvaluation || request.CooperativeWave64Workgroup) + { + error = "The Metal backend does not yet support native tessellation evaluation or cooperative wave64 workgroups."; + return false; + } try { BindingLayoutValidator.Validate( diff --git a/src/SharpEmu.ShaderCompiler.Metal/Gen5MslTranslator.cs b/src/SharpEmu.ShaderCompiler.Metal/Gen5MslTranslator.cs index 698c0d0..920e10b 100644 --- a/src/SharpEmu.ShaderCompiler.Metal/Gen5MslTranslator.cs +++ b/src/SharpEmu.ShaderCompiler.Metal/Gen5MslTranslator.cs @@ -914,6 +914,8 @@ public static partial class Gen5MslTranslator case "STtraceData": // Wave scheduling priority hint; no effect on results. case "SSetprio": + // Sleeps the wave for a number of clocks; no effect on results. + case "SSleep": case "SClause": case "VNop": // NGG shaders bracket their exports with s_sendmsg diff --git a/src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvShader.cs b/src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvShader.cs index 8eeb954..4aca043 100644 --- a/src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvShader.cs +++ b/src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvShader.cs @@ -13,6 +13,7 @@ public enum Gen5SpirvStage Vertex, Pixel, Compute, + TessellationEvaluation, } public sealed record Gen5SpirvShader( diff --git a/src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvTranslator.Alu.cs b/src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvTranslator.Alu.cs index bce01f0..a8a1068 100644 --- a/src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvTranslator.Alu.cs +++ b/src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvTranslator.Alu.cs @@ -1689,10 +1689,17 @@ public static partial class Gen5SpirvTranslator } // Packed f16 (VOP3P) arithmetic. Each source register holds two f16 values, - // one per result lane. Every f16<->f32 conversion is done with the explicit - // integer sequences below (EmitHalfToFloat / EmitFloatToHalf) instead of - // GLSL UnpackHalf2x16 / PackHalf2x16, whose subnormal and rounding behaviour - // is implementation-defined without float-controls execution modes. The two + // one per result lane. Every f16<->f32 conversion goes through + // EmitHalfToFloat / EmitFloatToHalf, which default to the explicit integer + // sequences below rather than GLSL UnpackHalf2x16 / PackHalf2x16, whose + // subnormal and rounding behaviour is implementation-defined without + // float-controls execution modes. The host switches those two helpers to the + // native ext instructions (ShaderCompileRequest.NativeHalfConversionExact) only + // after measuring them bit-exact against these sequences on the running device, + // because a device capability bit does not promise that: MoltenVK on Apple + // silicon reports shaderRoundingModeRTEFloat16 and + // shaderSignedZeroInfNanPreserveFloat16 yet not shaderDenormPreserveFloat16. + // The two // lanes are computed independently: each operand half is widened exactly to // f32, op_sel/op_sel_hi pick the source half and neg_lo/neg_hi negate it, the // op runs in f32, and the result is rounded back to f16 with round-to-nearest- @@ -1807,8 +1814,11 @@ public static partial class Gen5SpirvTranslator } // Reads one V_FMA_MIX source as an f32. op_sel_hi selects whether a - // register operand is taken as an f16 (the half picked by op_sel, widened - // exactly to f32) or as a full f32; inline constants are always f32. The + // register or literal operand is taken as an f16 (the half picked by op_sel, + // widened exactly to f32) or as a full f32; inline constants keep their value + // either way. A literal is an f16 bit pattern then: reading 0x34CD as an f32 + // made Silent Hill's luminance weights denormal zeros, and the lighting that + // divides by that luminance came out thousands of times too bright. The // per-operand neg_hi bit takes the absolute value and neg negates, in that // order (abs-then-neg), reusing the VOP3P modifier fields the way the mix // ops define them rather than the packed low/high-lane meaning. @@ -1820,7 +1830,7 @@ public static partial class Gen5SpirvTranslator var source = instruction.Sources[index]; var readAsHalf = ((control.OpSelHiMask >> index) & 1) != 0 && - source.Kind is Gen5OperandKind.VectorRegister or Gen5OperandKind.ScalarRegister; + source.Kind is Gen5OperandKind.VectorRegister or Gen5OperandKind.ScalarRegister or Gen5OperandKind.LiteralConstant; uint value; if (readAsHalf) @@ -1863,6 +1873,16 @@ public static partial class Gen5SpirvTranslator Gen5Vop3pControl control, bool highLane) { + if (instruction.Opcode == "VPkFmaF16" && _request.SupportsExactFloat16Conversions && + instruction.Sources.All(source => source.Kind != Gen5OperandKind.EncodedConstant)) + { + var select = highLane ? control.OpSelHiMask : control.OpSelMask; + var negate = highLane ? control.NegHiMask : control.NegLoMask; + uint Operand(int index) => ReadNativePackedHalf(GetRawSource(instruction, index), + ((select >> index) & 1) != 0, ((negate >> index) & 1) != 0); + return EmitNativePackedHalfFma(Operand(0), Operand(1), Operand(2), control.Clamp); + } + var left = EmitPackedF16Operand(instruction, control, 0, highLane); var right = EmitPackedF16Operand(instruction, control, 1, highLane); uint value; @@ -1909,6 +1929,12 @@ public static partial class Gen5SpirvTranslator uint accumulator, bool highLane) { + if (_request.SupportsExactFloat16Conversions) + return EmitNativePackedHalfFma( + ReadNativePackedHalf(GetRawSource(instruction, 0), highLane), + ReadNativePackedHalf(GetRawSource(instruction, 1), highLane), + ReadNativePackedHalf(accumulator, highLane), clamp: false); + var left = EmitPackedF16Half(GetRawSource(instruction, 0), highLane); var right = EmitPackedF16Half(GetRawSource(instruction, 1), highLane); var addend = EmitPackedF16Half(accumulator, highLane); @@ -1936,6 +1962,26 @@ public static partial class Gen5SpirvTranslator return Bitcast(_uintType, clamped); } + // Keep packed operands in half until the fused operation, avoiding widening, + // software NaN reconstruction and narrowing at every input. Inline f32 constants + // use the original path because they need not be exactly representable in half. + private uint ReadNativePackedHalf(uint raw, bool highLane, bool negate = false) + { + var bits = highLane ? ShiftRightLogical(raw, UInt(16)) : raw; + if (negate) bits = BitwiseXor(bits, UInt(0x8000)); + return _module.AddInstruction(SpirvOp.CompositeExtract, _halfType, Bitcast(_half2Type, bits), 0); + } + + private uint EmitNativePackedHalfFma(uint left, uint right, uint addend, bool clamp) + { + var fused = Ext(50, _halfType, left, right, addend); + _module.AddDecoration(fused, SpirvDecoration.NoContraction); + var bits = Bitcast(_uintType, _module.AddInstruction(SpirvOp.FConvert, _floatType, fused)); + if (clamp) bits = EmitClampToUnitInterval(bits); + // Preserve the common output NaN canonicalization and packing rules. + return EmitFloatToHalf(bits); + } + // Fused f16 multiply-add with a single rounding, emulated in f32 without the // Float16 capability. The f32 product of two widened f16 values is exact // (11-bit significands, and the exponent stays inside the f32 normal range: @@ -2014,11 +2060,12 @@ public static partial class Gen5SpirvTranslator bool highLane) { var operand = instruction.Sources[index]; - if (operand.Kind is Gen5OperandKind.EncodedConstant or Gen5OperandKind.LiteralConstant) + if (operand.Kind is Gen5OperandKind.EncodedConstant) { - // An inline constant or literal is a single f32 value, not a - // packed register: both lanes take the exactly-converted f16 of - // the same constant, so there is no half to select. + // An inline constant is replicated into both halves: both lanes take + // the exactly-converted f16 of the same constant, whatever op_sel says. + // A 32-bit literal is not: it holds two f16 halves that op_sel / + // op_sel_hi select like a register's, so it takes the path below. var constant = GetFloatSource(instruction, index); var constantNegate = highLane ? control.NegHiMask : control.NegLoMask; if (((constantNegate >> index) & 1) != 0) @@ -2067,8 +2114,23 @@ public static partial class Gen5SpirvTranslator // Widens an f16 value held in the low 16 bits of `halfBits` to an f32 bit // pattern, exactly (subnormals normalised, Inf/NaN and signed zero preserved). // Mirrors the branchless HalfToFloat reference validated against System.Half. + // With NativeHalfConversionExact the same value comes from GLSL UnpackHalf2x16, + // which the driver lowers to one hardware convert instead of ~20 integer ops. private uint EmitHalfToFloat(uint halfBits) { + if (_request.SupportsExactFloat16Conversions) + { + return EmitNativeHalfToFloat(halfBits); + } + + if (_nativeHalfConversionExact) + { + var unpacked = Ext(62, _vec2Type, BitwiseAnd(halfBits, UInt(0xFFFF))); + return Bitcast( + _uintType, + _module.AddInstruction(SpirvOp.CompositeExtract, _floatType, unpacked, 0)); + } + var sign = ShiftLeftLogical(BitwiseAnd(halfBits, UInt(0x8000)), UInt(16)); var exponent = BitwiseAnd(ShiftRightLogical(halfBits, UInt(10)), UInt(0x1F)); var mantissa = BitwiseAnd(halfBits, UInt(0x3FF)); @@ -2098,8 +2160,24 @@ public static partial class Gen5SpirvTranslator // Narrows an f32 bit pattern to an f16 value in the low 16 bits, rounding to // nearest even (subnormals, overflow-to-Inf and NaN/Inf handled). Mirrors the // branchless FloatToHalf reference validated exhaustively against System.Half. + // With NativeHalfConversionExact the same value comes from GLSL PackHalf2x16. private uint EmitFloatToHalf(uint bits) { + if (_request.SupportsExactFloat16Conversions) + { + return EmitNativeFloatToHalf(bits); + } + + if (_nativeHalfConversionExact) + { + var pair = _module.AddInstruction( + SpirvOp.CompositeConstruct, + _vec2Type, + Bitcast(_floatType, bits), + Float(0)); + return BitwiseAnd(Ext(58, _uintType, pair), UInt(0xFFFF)); + } + var sign = BitwiseAnd(ShiftRightLogical(bits, UInt(16)), UInt(0x8000)); var absolute = BitwiseAnd(bits, UInt(0x7FFF_FFFF)); @@ -2148,6 +2226,34 @@ public static partial class Gen5SpirvTranslator return SelectU(isInfinityNan, infinityNan, BitwiseOr(sign, finite)); } + // EmitHalfToFloat with a native conversion. Widening is exact; only NaNs are + // rebuilt, since drivers may quiet a signalling NaN where the sequence above keeps + // its payload bits as they are. + private uint EmitNativeHalfToFloat(uint halfBits) + { + var half = _module.AddInstruction( + SpirvOp.CompositeExtract, _halfType, Bitcast(_half2Type, halfBits), 0); + var widened = Bitcast(_uintType, _module.AddInstruction(SpirvOp.FConvert, _floatType, half)); + var magnitude = BitwiseAnd(halfBits, UInt(0x7FFF)); + var nan = BitwiseOr( + ShiftLeftLogical(BitwiseAnd(halfBits, UInt(0x8000)), UInt(16)), + BitwiseOr(UInt(0x7F80_0000), ShiftLeftLogical(BitwiseAnd(halfBits, UInt(0x3FF)), UInt(13)))); + return SelectU(UCmp(SpirvOp.UGreaterThan, magnitude, UInt(0x7C00)), nan, widened); + } + + // EmitFloatToHalf with a native conversion, rounded to nearest even by the + // module's RoundingModeRTE. A NaN becomes the same quiet NaN as above (sign kept, + // payload dropped) rather than the driver's canonical NaN. + private uint EmitNativeFloatToHalf(uint bits) + { + var half = _module.AddInstruction(SpirvOp.FConvert, _halfType, Bitcast(_floatType, bits)); + var pair = _module.AddInstruction(SpirvOp.CompositeConstruct, _half2Type, half, half); + var narrowed = BitwiseAnd(Bitcast(_uintType, pair), UInt(0xFFFF)); + var nan = BitwiseOr(BitwiseAnd(ShiftRightLogical(bits, UInt(16)), UInt(0x8000)), UInt(0x7E00)); + var isNan = UCmp(SpirvOp.UGreaterThan, BitwiseAnd(bits, UInt(0x7FFF_FFFF)), UInt(0x7F80_0000)); + return SelectU(isNan, nan, narrowed); + } + private uint SelectU(uint condition, uint whenTrue, uint whenFalse) => _module.AddInstruction(SpirvOp.Select, _uintType, condition, whenTrue, whenFalse); @@ -2717,40 +2823,19 @@ public static partial class Gen5SpirvTranslator if (instruction.Opcode == "SFlbitI32B64") { - var wide = GetRawSource64(instruction, 0); - var low = _module.AddInstruction( - SpirvOp.UConvert, - _uintType, - wide); - var high = _module.AddInstruction( - SpirvOp.UConvert, - _uintType, - ShiftRightLogical64( - wide, - _module.Constant64(_ulongType, 32))); - var highClz = _module.AddInstruction( - SpirvOp.ISub, - _uintType, - UInt(31), - Ext(74, _uintType, high)); - var lowClz = _module.AddInstruction( - SpirvOp.ISub, - _uintType, - UInt(31), - Ext(74, _uintType, low)); - var lowResult = IAdd(lowClz, UInt(32)); - var leadingZeroResult = _module.AddInstruction( - SpirvOp.Select, - _uintType, - IsNotZero(high), - highClz, - _module.AddInstruction( - SpirvOp.Select, - _uintType, - IsNotZero(low), - lowResult, - UInt(uint.MaxValue))); - StoreS(destination, leadingZeroResult); + var source = GetRawSource64(instruction, 0); + var low = _module.AddInstruction(SpirvOp.UConvert, _uintType, source); + var high = _module.AddInstruction(SpirvOp.UConvert, _uintType, + ShiftRightLogical64(source, ULong(32))); + var highIndex = _module.AddInstruction(SpirvOp.ISub, _uintType, + UInt(31), Ext(75, _uintType, high)); + var lowIndex = _module.AddInstruction(SpirvOp.ISub, _uintType, + UInt(63), Ext(75, _uintType, low)); + var bitIndex = _module.AddInstruction(SpirvOp.Select, _uintType, + IsNotZero(high), highIndex, + _module.AddInstruction(SpirvOp.Select, _uintType, + IsNotZero(low), lowIndex, UInt(uint.MaxValue))); + StoreS(destination, bitIndex); return true; } @@ -2764,13 +2849,25 @@ public static partial class Gen5SpirvTranslator var left = GetRawSource(instruction, 0); if (instruction.Opcode == "SBitreplicateB64B32") { - // S_BITREPLICATE_B64_B32 broadcasts the source dword into both - // halves of its 64-bit SGPR destination and does not update SCC. - StoreS(destination, left); - StoreS(destination + 1, left); + // S_BITREPLICATE_B64_B32 doubles every source bit: bit i lands in bits + // 2i and 2i+1 of the 64-bit destination, so the low dword comes from + // source bits 0-15 and the high dword from bits 16-31. SCC is unchanged. + StoreS(destination, ReplicateBits(BitwiseAnd(left, UInt(0xFFFF)))); + StoreS(destination + 1, ReplicateBits(ShiftRightLogical(left, UInt(16)))); return true; } + // Spreads the low 16 bits of `value` to the even bit positions and copies each + // into the odd position above it: abcd -> aabbccdd. + uint ReplicateBits(uint value) + { + value = BitwiseAnd(BitwiseOr(value, ShiftLeftLogical(value, UInt(8))), UInt(0x00FF_00FF)); + value = BitwiseAnd(BitwiseOr(value, ShiftLeftLogical(value, UInt(4))), UInt(0x0F0F_0F0F)); + value = BitwiseAnd(BitwiseOr(value, ShiftLeftLogical(value, UInt(2))), UInt(0x3333_3333)); + value = BitwiseAnd(BitwiseOr(value, ShiftLeftLogical(value, UInt(1))), UInt(0x5555_5555)); + return BitwiseOr(value, ShiftLeftLogical(value, UInt(1))); + } + if (instruction.Opcode.EndsWith("SaveexecB32", StringComparison.Ordinal)) { var oldExec64 = BooleanToWaveMask(Load(_boolType, _exec)); @@ -2878,7 +2975,7 @@ public static partial class Gen5SpirvTranslator case "SFlbitI32B32": { // Count leading zero bits, 0xFFFFFFFF when the source is zero. - var msb = Ext(74, _uintType, left); + var msb = Ext(75, _uintType, left); var clz = _module.AddInstruction(SpirvOp.ISub, _uintType, UInt(31), msb); result = _module.AddInstruction( SpirvOp.Select, _uintType, IsNotZero(left), clz, UInt(0xFFFFFFFFu)); @@ -3953,7 +4050,7 @@ public static partial class Gen5SpirvTranslator { targetLane = BitwiseAnd(targetLane, UInt(31)); } - var shuffled = ShuffleLane(value, targetLane); + var shuffled = ShuffleGuestLaneOrSelf(value, targetLane, out var sourcePresent); if (control.FetchInactive) { return shuffled; @@ -3966,11 +4063,15 @@ public static partial class Gen5SpirvTranslator UInt(1), UInt(0)); var sourceActive = IsNotZero( - ShuffleLane(activeWord, targetLane)); + ShuffleGuestLaneOrSelf(activeWord, targetLane, out _)); return _module.AddInstruction( SpirvOp.Select, _uintType, - sourceActive, + _module.AddInstruction( + SpirvOp.LogicalOr, + _boolType, + LogicalNot(sourcePresent), + sourceActive), shuffled, UInt(0)); } @@ -3989,7 +4090,7 @@ public static partial class Gen5SpirvTranslator // operations are limited to a single half-wave for some encodings, so we // must not clamp wave64 lanes to 31; use the full lane mask instead. safeTarget = BitwiseAnd(safeTarget, UInt(_waveLaneCount == 64 ? 63u : 31u)); - var shuffled = ShuffleLane(value, safeTarget); + var shuffled = ShuffleGuestLaneOrSelf(value, safeTarget, out var sourcePresent); var sourceAvailable = inRange; if (!control.FetchInactive) @@ -4000,12 +4101,16 @@ public static partial class Gen5SpirvTranslator Load(_boolType, _exec), UInt(1), UInt(0)); - var shuffledActive = ShuffleLane(activeWord, safeTarget); + var shuffledActive = ShuffleGuestLaneOrSelf(activeWord, safeTarget, out _); sourceAvailable = _module.AddInstruction( SpirvOp.LogicalAnd, _boolType, sourceAvailable, - IsNotZero(shuffledActive)); + _module.AddInstruction( + SpirvOp.LogicalOr, + _boolType, + LogicalNot(sourcePresent), + IsNotZero(shuffledActive))); } return _module.AddInstruction( @@ -5038,7 +5143,14 @@ public static partial class Gen5SpirvTranslator var sourceValue = GetRawSource(instruction, 0); var selectedLane = BitwiseAnd(GetRawSource(instruction, 1), UInt(LaneSelectMask)); - if (_emulateWave64) + if (_emulateWave64 && instruction.Sources[0].Kind == Gen5OperandKind.VectorRegister && + TryGetConstantLane(instruction, out var spillLane) && + _laneSpillSlots.TryGetValue((instruction.Sources[0].Value, spillLane), out var spillSlot)) + { + // A pure scalar spill slot: every invocation already holds the wave-uniform value. + StoreS(destination, Load(_uintType, spillSlot)); + } + else if (_emulateWave64) { // The selected guest lane can belong to another host subgroup. // Read it even when the guest execution mask disables that lane. @@ -5054,6 +5166,13 @@ public static partial class Gen5SpirvTranslator selectedLane); StoreS(destination, ReadLaneSpillSlot(instruction, selectedLane, broadcast)); } + else if (_subgroupInvocationIdInput != 0) + { + // A graphics wave is a group of up to 32 host lanes (GuestWaveLane). Read the + // selected lane of the invocation's own group, whatever its EXEC state. + var read = ShuffleGuestLaneOrSelf(sourceValue, selectedLane, out _); + StoreS(destination, ReadLaneSpillSlot(instruction, selectedLane, read)); + } else { // Fallback: no subgroup ops, read current lane's value, or the @@ -5115,11 +5234,9 @@ public static partial class Gen5SpirvTranslator } var targetLane = IAdd(rowBase, selector); - // Mask to guest wave size — on Radeon hardware DPP is limited to a - // single half-wave for some encodings, but we must not clamp wave64 - // lanes to 31; use the full lane mask instead. + // Select a guest lane first; ShuffleGuestLaneOrSelf maps it to the host subgroup. targetLane = BitwiseAnd(targetLane, UInt(_waveLaneCount == 64 ? 63u : 31u)); - var shuffled = ShuffleLane(value, targetLane); + var shuffled = ShuffleGuestLaneOrSelf(value, targetLane, out var sourcePresent); var fetchInactive = (control.OperandSelect & 1) != 0; if (fetchInactive) { @@ -5133,11 +5250,15 @@ public static partial class Gen5SpirvTranslator UInt(1), UInt(0)); var sourceActive = IsNotZero( - ShuffleLane(activeWord, targetLane)); + ShuffleGuestLaneOrSelf(activeWord, targetLane, out _)); return _module.AddInstruction( SpirvOp.Select, _uintType, - sourceActive, + _module.AddInstruction( + SpirvOp.LogicalOr, + _boolType, + LogicalNot(sourcePresent), + sourceActive), shuffled, UInt(0)); } @@ -5228,12 +5349,33 @@ public static partial class Gen5SpirvTranslator return Bitcast(_uintType, value); } + // Rounds an f32 toward zero onto an f16 value, so packHalf2x16 then encodes it exactly + // (V_CVT_PKRTZ_F16_F32). Toward zero, a finite value beyond the f16 range becomes + // +-65504, never infinity: Silent Hill's deferred lighting packs values past 65504 and + // its fog pass then scales them down, while an infinity stays infinite, turns into NaN + // and blacks out the whole title-menu scene. Subnormal results keep whole 2^-24 steps. private uint TruncateFloat32ForPack(uint value) { - var raw = BitwiseAnd( - Bitcast(_uintType, value), - UInt(0xFFFF_E000)); - return Bitcast(_floatType, raw); + var normal = Bitcast( + _floatType, + BitwiseAnd(Bitcast(_uintType, Ext(43, _floatType, value, Float(-65504f), Float(65504f))), UInt(0xFFFF_E000))); + var subnormal = _module.AddInstruction( + SpirvOp.FMul, + _floatType, + Ext(3, _floatType, _module.AddInstruction(SpirvOp.FMul, _floatType, value, Float(16777216f))), + Float(1f / 16777216f)); + var isSubnormal = _module.AddInstruction( + SpirvOp.FOrdLessThan, + _boolType, + Ext(4, _floatType, value), + Float(6.10351562e-05f)); + var finite = _module.AddInstruction(SpirvOp.Select, _floatType, isSubnormal, subnormal, normal); + var special = _module.AddInstruction( + SpirvOp.LogicalOr, + _boolType, + _module.AddInstruction(SpirvOp.IsNan, _boolType, value), + _module.AddInstruction(SpirvOp.IsInf, _boolType, value)); + return _module.AddInstruction(SpirvOp.Select, _floatType, special, value, finite); } private uint Ext(uint operation, uint resultType, params uint[] operands) diff --git a/src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvTranslator.CooperativeWaves.cs b/src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvTranslator.CooperativeWaves.cs new file mode 100644 index 0000000..c5bb04f --- /dev/null +++ b/src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvTranslator.CooperativeWaves.cs @@ -0,0 +1,113 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +namespace SharpEmu.ShaderCompiler.Vulkan; + +public static partial class Gen5SpirvTranslator +{ + private sealed partial class CompilationContext + { + private uint _cooperativeProgramCounters; + // Nonzero only while emitting a block selected by the workgroup dispatcher. + private uint _executingWave; + private uint CooperativeWaveCount => checked(_localSizeX * _localSizeY * _localSizeZ / 64); + private uint CooperativeWaveId() => ShiftRightLogical(Load(_uintType, _localInvocationIndexInput), UInt(6)); + + private void DeclareCooperativeWaveScratch() + { + var array = _module.TypeArray(_uintType, CooperativeWaveCount); + _cooperativeProgramCounters = _module.AddGlobalVariable( + _module.TypePointer(SpirvStorageClass.Workgroup, array), SpirvStorageClass.Workgroup); + _module.AddName(_cooperativeProgramCounters, "waveProgramCounters"); + _interfaces.Add(_cooperativeProgramCounters); + } + + private uint CooperativeCounterPointer(uint index) => _module.AddInstruction( + SpirvOp.AccessChain, _wave64ExchangeElementPointer, _cooperativeProgramCounters, index); + + private uint SelectCooperativeBlock(IReadOnlyList blocks) + { + var pc = _module.AddInstruction(SpirvOp.Select, _uintType, + Load(_boolType, _programActive), Load(_uintType, _programCounter), UInt(uint.MaxValue)); + // Waiting guest waves must not pass a guest barrier until all still + // running waves arrive. Host rendezvous used for ballots are distinct + // from these guest barriers. + for (var block = 0; block < blocks.Count; block++) + { + if (_request.Program.Instructions[blocks[block].StartIndex].Opcode != "SBarrier") continue; + pc = _module.AddInstruction(SpirvOp.Select, _uintType, + _module.AddInstruction(SpirvOp.IEqual, _boolType, pc, UInt((uint)block)), + UInt((uint)block | 0x80000000u), pc); + } + EmitConditional(_module.AddInstruction(SpirvOp.IEqual, _boolType, GuestWaveLane(), UInt(0)), + () => Store(CooperativeCounterPointer(CooperativeWaveId()), pc)); + EmitWave64Barrier(); + var selected = Load(_uintType, CooperativeCounterPointer(UInt(0))); + for (uint wave = 1; wave < CooperativeWaveCount; wave++) + { + var candidate = Load(_uintType, CooperativeCounterPointer(UInt(wave))); + selected = _module.AddInstruction(SpirvOp.Select, _uintType, + _module.AddInstruction(SpirvOp.ULessThan, _boolType, candidate, selected), candidate, selected); + } + // Every invocation finishes reading before a later iteration overwrites the counters. + EmitWave64Barrier(); + return _module.AddInstruction(SpirvOp.Select, _uintType, + _module.AddInstruction(SpirvOp.IEqual, _boolType, selected, UInt(uint.MaxValue)), + selected, BitwiseAnd(selected, UInt(0x7FFFFFFF))); + } + + private bool TryEmitCooperativeDispatcher(IReadOnlyList blocks, out string error) + { + error = string.Empty; + var header = _module.AllocateId(); + var condition = _module.AllocateId(); + var body = _module.AllocateId(); + var continuation = _module.AllocateId(); + var exit = _module.AllocateId(); + _module.AddStatement(SpirvOp.Branch, header); + _module.AddLabel(header); + _module.AddStatement(SpirvOp.LoopMerge, exit, continuation, 0); + _module.AddStatement(SpirvOp.Branch, condition); + _module.AddLabel(condition); + var selector = SelectCooperativeBlock(blocks); + var anyActive = _module.AddInstruction(SpirvOp.INotEqual, _boolType, selector, UInt(uint.MaxValue)); + _module.AddStatement(SpirvOp.BranchConditional, anyActive, body, exit); + _module.AddLabel(body); + var merge = _module.AllocateId(); + var invalid = _module.AllocateId(); + var labels = blocks.Select(_ => _module.AllocateId()).ToArray(); + var operands = new List { selector, invalid }; + for (var block = 0; block < blocks.Count; block++) + { + operands.Add((uint)block); operands.Add(labels[block]); + } + _module.AddStatement(SpirvOp.SelectionMerge, merge, 0); + _module.AddStatement(SpirvOp.Switch, operands.ToArray()); + for (var block = 0; block < blocks.Count; block++) + { + _module.AddLabel(labels[block]); + _executingWave = LogicalAnd(Load(_boolType, _programActive), + _module.AddInstruction(SpirvOp.IEqual, _boolType, Load(_uintType, _programCounter), UInt((uint)block))); + Store(_exec, Load(_boolType, _exec)); + // All host subgroups execute the same block, including its rendezvous; + // only the guest waves at this PC may change architectural state. + if (!TryEmitBlock(blocks, block, out error)) return false; + _executingWave = 0; + Store(_exec, IsLaneSetInMaskRegisters(126)); + _module.AddStatement(SpirvOp.Branch, merge); + } + _module.AddLabel(invalid); + Store(_programActive, _module.ConstantBool(false)); + _module.AddStatement(SpirvOp.Branch, merge); + _module.AddLabel(merge); + _module.AddStatement(SpirvOp.Branch, continuation); + _module.AddLabel(continuation); + _module.AddStatement(SpirvOp.Branch, header); + _module.AddLabel(exit); + return true; + } + + private uint GuardCooperativeUint(uint value, uint previous) => _executingWave == 0 ? value : + _module.AddInstruction(SpirvOp.Select, _uintType, _executingWave, value, previous); + } +} diff --git a/src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvTranslator.Interpolation.cs b/src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvTranslator.Interpolation.cs index 049d1f2..4572570 100644 --- a/src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvTranslator.Interpolation.cs +++ b/src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvTranslator.Interpolation.cs @@ -16,6 +16,7 @@ public static partial class Gen5SpirvTranslator private uint _perspectiveBarycentric; private readonly Dictionary _barycentricInputs = []; private uint _interpolationSampleId; + private readonly HashSet _fixedSampleInterpolants = []; private uint _frontFacingInput; private uint _ancillaryLayerInput; private uint _sampleMaskInput; @@ -23,8 +24,36 @@ public static partial class Gen5SpirvTranslator private const uint InterpolateAtSample = 77; private const uint InterpolateAtOffset = 78; + private void DecorateSampleInterpolant(uint variable) + { + // Only a single active interpolation mode establishes one sample position + // for ordinary VINTRP inputs. Mixed modes require source-aware lowering. + var modes = _pixelInputEnable & _pixelInputAddress & 0x7Fu; + if (modes is not (0x1u or 0x10u)) + return; + _module.AddCapability(SpirvCapability.SampleRateShading); + if (modes == 0x10u) + _module.AddDecoration(variable, SpirvDecoration.NoPerspective); + if (_request.PixelInterpolationSample.HasValue) + { + _module.AddCapability(SpirvCapability.InterpolationFunction); + _fixedSampleInterpolants.Add(variable); + } + else _module.AddDecoration(variable, SpirvDecoration.Sample); + } + private void DeclareInterpolationParameters() { + var offsetCount = _request.PixelInterpolationSample.HasValue ? 4u : 4u * _request.PixelRasterizationSamples; + if (_request.PixelCustomSampleOffsets.Count != 0 && + (_request.PixelCustomSampleOffsets.Count != offsetCount || + (!_request.PixelInterpolationSample.HasValue && + (_request.PixelRasterizationSamples is < 2 or > 16 || + (_request.PixelRasterizationSamples & (_request.PixelRasterizationSamples - 1)) != 0)) || + _request.PixelCustomSampleOffsets.Any(offset => !float.IsFinite(offset.X) || !float.IsFinite(offset.Y) || + offset.X < -.5f || offset.X > .5f || offset.Y < -.5f || offset.Y > .5f))) + throw new NotSupportedException("Custom-sample interpolation requires four finite offsets per selected sample within the pixel."); + foreach (var instruction in _request.Program.Instructions) { if (instruction.Opcode == "VInterpMovF32" && @@ -92,7 +121,7 @@ public static partial class Gen5SpirvTranslator _barycentricInputs.Add(bit, variable); } - if ((enabledInputs & 0x11u) != 0) + if ((enabledInputs & 0x11u) != 0 && !_request.PixelInterpolationSample.HasValue) { _module.AddCapability(SpirvCapability.SampleRateShading); _interpolationSampleId = _module.AddGlobalVariable( @@ -183,10 +212,12 @@ public static partial class Gen5SpirvTranslator vgpr++; } - // The perspective barycentrics a smooth slot sharing a per-vertex input is rebuilt with. + // Barycentrics used to reconstruct a smooth slot sharing a per-vertex input. private void DeclarePerspectiveBarycentric() { - foreach (var bit in new[] { 0, 1, 2 }) + var modes = _pixelInputEnable & _pixelInputAddress & 0x7Fu; + var linear = (modes & 0x70u) != 0 && (modes & 0x7u) == 0; + foreach (var bit in linear ? new[] { 4, 5, 6 } : new[] { 0, 1, 2 }) { if (_barycentricInputs.TryGetValue(bit, out var existing)) { @@ -197,7 +228,8 @@ public static partial class Gen5SpirvTranslator _perspectiveBarycentric = _module.AddGlobalVariable( _module.TypePointer(SpirvStorageClass.Input, _vec3Type), SpirvStorageClass.Input); - _module.AddDecoration(_perspectiveBarycentric, SpirvDecoration.BuiltIn, (uint)SpirvBuiltIn.BaryCoordKhr); + _module.AddDecoration(_perspectiveBarycentric, SpirvDecoration.BuiltIn, + (uint)(linear ? SpirvBuiltIn.BaryCoordNoPerspKhr : SpirvBuiltIn.BaryCoordKhr)); _interfaces.Add(_perspectiveBarycentric); } @@ -216,7 +248,10 @@ public static partial class Gen5SpirvTranslator var value = LoadVertex(0); if (!flat) { - var barycentric = Load(_vec3Type, _perspectiveBarycentric); + var modes = _pixelInputEnable & _pixelInputAddress & 0x7Fu; + var bit = modes switch { 1u => 0, 2u => 1, 4u => 2, 16u => 4, 32u => 5, 64u => 6, _ => -1 }; + var barycentric = bit >= 0 ? LoadBarycentricCoordinates(bit, _perspectiveBarycentric) + : Load(_vec3Type, _perspectiveBarycentric); value = _module.AddInstruction(SpirvOp.FMul, _floatType, value, _module.AddInstruction(SpirvOp.CompositeExtract, _floatType, barycentric, 0)); for (uint vertex = 1; vertex < 3; vertex++) @@ -230,10 +265,85 @@ public static partial class Gen5SpirvTranslator StoreV(destination, Bitcast(_uintType, value)); } + private void DeclareFrontFaceInput() + { + if ((_pixelInputAddress & _pixelInputEnable & (1u << 12)) == 0) + { + return; + } + + _frontFacingInput = _module.AddGlobalVariable( + _module.TypePointer(SpirvStorageClass.Input, _boolType), SpirvStorageClass.Input); + _module.AddDecoration(_frontFacingInput, SpirvDecoration.BuiltIn, (uint)SpirvBuiltIn.FrontFacing); + _interfaces.Add(_frontFacingInput); + } + + private void EmitFrontFaceInput(ref uint vgpr) + { + if ((_pixelInputAddress & (1u << 12)) == 0) + { + return; + } + + if (_frontFacingInput != 0) + { + // The guest register contains the bits of 1.0f for a front face and zero for a back face. + var value = _module.AddInstruction(SpirvOp.Select, _uintType, + Load(_boolType, _frontFacingInput), UInt(0x3F800000u), UInt(0)); + StoreV(vgpr, value, guardWithExec: false); + } + + vgpr++; + } + + private uint InterpolationSampleIndex() => _request.PixelInterpolationSample is uint sample + ? _module.Constant(_intType, sample) + : Load(_intType, _interpolationSampleId); + + private uint ExplicitSampleInterpolation(uint type, uint variable) + { + if (_request.PixelCustomSampleOffsets.Count == 0) + return _module.AddInstruction(SpirvOp.ExtInst, type, _glsl, InterpolateAtSample, variable, InterpolationSampleIndex()); + var coord = Load(_vec4Type, _fragCoordInput); + uint Odd(int component) + { + var integer = _module.AddInstruction(SpirvOp.ConvertFToU, _uintType, + _module.AddInstruction(SpirvOp.CompositeExtract, _floatType, coord, (uint)component)); + return _module.AddInstruction(SpirvOp.INotEqual, _boolType, + _module.AddInstruction(SpirvOp.BitwiseAnd, _uintType, integer, UInt(1)), UInt(0)); + } + var samplesPerPixel = _request.PixelInterpolationSample.HasValue ? 1u : _request.PixelRasterizationSamples; + var sampleId = _request.PixelInterpolationSample.HasValue ? UInt(0) : + Bitcast(_uintType, Load(_intType, _interpolationSampleId)); + uint Offset(int pixel) + { + uint At(uint sample) + { + var value = _request.PixelCustomSampleOffsets[(int)(pixel * samplesPerPixel + sample)]; + return _module.ConstantComposite(_vec2Type, Float(value.X), Float(value.Y)); + } + var selected = At(0); + for (uint sample = 1; sample < samplesPerPixel; sample++) + { + var matches = _module.AddInstruction(SpirvOp.IEqual, _boolType, sampleId, UInt(sample)); + selected = _module.AddInstruction(SpirvOp.Select, _vec2Type, matches, At(sample), selected); + } + return selected; + } + var xOdd = Odd(0); + var row0 = _module.AddInstruction(SpirvOp.Select, _vec2Type, xOdd, Offset(1), Offset(0)); + var row1 = _module.AddInstruction(SpirvOp.Select, _vec2Type, xOdd, Offset(3), Offset(2)); + var offset = _module.AddInstruction(SpirvOp.Select, _vec2Type, Odd(1), row1, row0); + return _module.AddInstruction(SpirvOp.ExtInst, type, _glsl, InterpolateAtOffset, variable, offset); + } + + private uint LoadOrdinaryInterpolant(uint variable) => _fixedSampleInterpolants.Contains(variable) + ? ExplicitSampleInterpolation(_vec4Type, variable) + : Load(_vec4Type, variable); + private uint LoadBarycentricCoordinates(int bit, uint variable) => bit switch { - 0 or 4 => _module.AddInstruction(SpirvOp.ExtInst, _vec3Type, _glsl, InterpolateAtSample, - variable, Load(_intType, _interpolationSampleId)), + 0 or 4 => ExplicitSampleInterpolation(_vec3Type, variable), 2 or 6 => _module.AddInstruction(SpirvOp.ExtInst, _vec3Type, _glsl, InterpolateAtCentroid, variable), _ => _module.AddInstruction(SpirvOp.ExtInst, _vec3Type, _glsl, InterpolateAtOffset, variable, _module.ConstantNull(_vec2Type)), diff --git a/src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvTranslator.RenderScale.cs b/src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvTranslator.RenderScale.cs new file mode 100644 index 0000000..0648de1 --- /dev/null +++ b/src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvTranslator.RenderScale.cs @@ -0,0 +1,96 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.ShaderCompiler; + +namespace SharpEmu.ShaderCompiler.Vulkan; + +// Internal resolution scaling. The host may hold a guest render target at another size, so +// everything the program expresses in pixels has to be translated: its own pixel position, +// the integer coordinates it addresses an image with and the sizes a resource query reports. +// Which images are scaled is a per-draw fact, so the factors arrive in shader data rather +// than through a permutation. +public static partial class Gen5SpirvTranslator +{ + private sealed partial class CompilationContext + { + // Shader-data dwords, relative to BindingLayout.RenderScaleDword. + private const uint ImageCoordinateFactorDword = 0; + private const uint ImageSizeFactorDword = 1; + private const uint ScaledImageMaskLowDword = 2; + private const uint ScaledImageMaskHighDword = 3; + private const uint PixelPositionFactorDword = 4; + + private bool UsesRenderScale => _request.Bindings.UsesRenderScale; + + private uint LoadRenderScaleFloat(uint dword) => + Bitcast(_floatType, LoadShaderDataDword(UInt(_request.Bindings.RenderScaleDword + dword))); + + // One when the host holds this image at guest resolution, the scale otherwise. The + // mask is loaded per use so the value never has to dominate a later block. + private uint LoadImageScaleFactor(in SpirvImageResource resource, uint factorDword) + { + var index = resource.ResourceIndex; + var maskDword = index < 32 ? ScaledImageMaskLowDword : ScaledImageMaskHighDword; + var bit = UInt(1u << (int)(index & 31)); + var mask = LoadShaderDataDword(UInt(_request.Bindings.RenderScaleDword + maskDword)); + var scaled = _module.AddInstruction(SpirvOp.INotEqual, _boolType, + _module.AddInstruction(SpirvOp.BitwiseAnd, _uintType, mask, bit), UInt(0)); + return _module.AddInstruction(SpirvOp.Select, _floatType, scaled, LoadRenderScaleFloat(factorDword), Float(1f)); + } + + // The pixel position addresses host texels; dividing x and y by the attachments' + // factor hands the program the guest pixel it would have shaded natively. + private uint ScalePixelPosition(uint fragCoord) + { + if (!UsesRenderScale) + { + return fragCoord; + } + + var factor = LoadRenderScaleFloat(PixelPositionFactorDword); + var components = new uint[4]; + for (uint component = 0; component < 4; component++) + { + var value = _module.AddInstruction(SpirvOp.CompositeExtract, _floatType, fragCoord, component); + components[component] = component < 2 + ? _module.AddInstruction(SpirvOp.FMul, _floatType, value, factor) + : value; + } + + return _module.AddInstruction(SpirvOp.CompositeConstruct, _vec4Type, components); + } + + // A guest pixel coordinate in host texels. The program computed it against the guest + // size, so it is multiplied by the factor and truncated back to a texel index. + private uint ScaleIntegerImageCoordinate(uint coordinate, in SpirvImageResource resource) + { + if (!UsesRenderScale) + { + return coordinate; + } + + var factor = LoadImageScaleFactor(resource, ImageCoordinateFactorDword); + var scaled = _module.AddInstruction(SpirvOp.FMul, _floatType, + _module.AddInstruction(SpirvOp.ConvertSToF, _floatType, coordinate), factor); + return _module.AddInstruction(SpirvOp.ConvertFToS, _intType, scaled); + } + + // A resource query reports the size the game declared, not the host's: the program + // derives texture coordinates and loop bounds from it in guest pixels. + private uint ScaleImageQuerySize(uint size, in SpirvImageResource resource) + { + if (!UsesRenderScale) + { + return size; + } + + var factor = LoadImageScaleFactor(resource, ImageSizeFactorDword); + var scaled = _module.AddInstruction(SpirvOp.FAdd, _floatType, + _module.AddInstruction(SpirvOp.FMul, _floatType, + _module.AddInstruction(SpirvOp.ConvertSToF, _floatType, size), factor), + Float(0.5f)); + return _module.AddInstruction(SpirvOp.ConvertFToS, _intType, scaled); + } + } +} diff --git a/src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvTranslator.Resources.cs b/src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvTranslator.Resources.cs index 9d682a8..6626bf8 100644 --- a/src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvTranslator.Resources.cs +++ b/src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvTranslator.Resources.cs @@ -18,6 +18,27 @@ public static partial class Gen5SpirvTranslator public static bool TryCompileProgram(ShaderCompileRequest request, out Gen5SpirvShader shader, out string error) { shader = default!; + if (request.CooperativeWave64Workgroup && (request.Stage != ShaderStage.Compute || request.WaveSize != 64 || + (ulong)request.LocalSizeX * request.LocalSizeY * request.LocalSizeZ is 0 or > 256 || + (request.LocalSizeX * request.LocalSizeY * request.LocalSizeZ) % 64 != 0)) + { + error = "Cooperative wave64 requires a compute workgroup containing one to four complete waves."; + return false; + } + if (request.TessellationHull is not null && + (request.Stage != ShaderStage.Compute || !request.CooperativeWave64Workgroup || !request.Bindings.UsesTessellationData)) + { + error = "A merged hull program requires cooperative compute execution and tessellation runtime data."; + return false; + } + if (request.Stage == ShaderStage.TessellationEvaluation && + (request.Tessellation is not { } tessellation || + !Enum.IsDefined(tessellation.Domain) || !Enum.IsDefined(tessellation.Spacing) || + tessellation.Spacing == Gen5TessellationSpacing.PowerOfTwo)) + { + error = "Tessellation evaluation requires a supported tessellation configuration; power-of-two partitioning needs factor conversion."; + return false; + } try { BindingLayoutValidator.Validate( @@ -44,16 +65,20 @@ public static partial class Gen5SpirvTranslator return false; } + // A dual-source second output shares location 0 with index 1; every other output + // has index 0 and its own location. + var targets = request.PixelOutputs.Count(output => output.Index == 0); if (request.PixelOutputs.Select(output => output.GuestSlot).Distinct().Count() != request.PixelOutputs.Count || - request.PixelOutputs.Select(output => output.HostLocation).Distinct().Count() != request.PixelOutputs.Count) + request.PixelOutputs.Select(output => (output.HostLocation, output.Index)).Distinct().Count() != request.PixelOutputs.Count) { error = "pixel output guest slots and host locations must be unique"; return false; } - if (request.PixelOutputs.Any(output => output.HostLocation >= request.PixelOutputs.Count)) + if (request.PixelOutputs.Any(output => output.HostLocation >= targets || output.Index > 1 || + (output.Index == 1 && (output.HostLocation != 0 || targets != 1)))) { - error = "pixel output host locations must be dense in the 0..N-1 range"; + error = "pixel output host locations must be dense in the 0..N-1 range, with a second source only at location 0 of one target"; return false; } } @@ -69,17 +94,42 @@ public static partial class Gen5SpirvTranslator private uint _pushData; private uint _shaderData; private uint _flattenedTable; + private uint _imageSlotTableDwordCount; private uint _pageTable; private uint _faultBuffer; + private uint _runtimeDescriptorTable; + private uint _runtimeDescriptorMisses; + private uint _runtimeSamplerArray; + private uint _runtimeSamplerPointer; + private readonly Dictionary<(ImageDimension Dimension, ImageNumericClass NumericClass), + (uint Variable, uint ImageType, uint ElementPointer, uint ComponentType, uint VectorType, + ImageComponentKind Kind, SpirvImageDim Dimension, bool Arrayed)> _runtimeImageArrays = []; + private uint _runtimeImageNumericClassTable; private uint _globalDataShare; private uint _samplerArray; private uint _samplerType; private uint _samplerPointer; private uint _storageUlongPointer; private uint _physicalUintPointer; - private uint _deviceEntryScratch; private uint _deviceWordScratch; private uint _deviceBufferWordScratch; + + // The missing pages an invocation met, as one word of the fault bitmap and its bits. + // A page walk only merges into them; the invocation publishes them once when it ends + // (FlushPendingDeviceFaults). An atomic per page walk made the driver compile shaders + // with many device accesses for tens of seconds. Faults are residency hints read + // after the dispatch: a page in a second bitmap word is reported by a later dispatch, + // once the pages of the first word are resident and no longer fault. + private uint _pendingFaultWord; + private uint _pendingFaultBits; + private uint _pendingWrittenWord; + private uint _pendingWrittenBits; + + // The element a typed device-address store writes, chosen by the runtime format: + // a value and a mask for each of its (at most four) dwords, and its byte count. + private readonly uint[] _deviceStoreValueScratch = new uint[4]; + private readonly uint[] _deviceStoreMaskScratch = new uint[4]; + private uint _deviceStoreBytesScratch; private uint _pushDataBlockPointer; private uint _wordRuntimeArray; private uint _addressRuntimeArray; @@ -95,7 +145,8 @@ public static partial class Gen5SpirvTranslator bool Cube, bool Multisampled, SpirvImageDim Dimension, - IReadOnlyList Resources); + IReadOnlyList Resources, + uint SlotOffset); public CompilationContext(ShaderCompileRequest request) { @@ -104,6 +155,7 @@ public static partial class Gen5SpirvTranslator { ShaderStage.Vertex => Gen5SpirvStage.Vertex, ShaderStage.Pixel => Gen5SpirvStage.Pixel, + ShaderStage.TessellationEvaluation => Gen5SpirvStage.TessellationEvaluation, _ => Gen5SpirvStage.Compute, }; _pixelOutputBindings = request.PixelOutputs; @@ -111,12 +163,15 @@ public static partial class Gen5SpirvTranslator _stage == Gen5SpirvStage.Pixel && request.Program.Instructions.Any(static instruction => instruction.Control is Gen5ExportControl { ValidMask: true }); _enableGraphicsSubgroupOperations = _stage == Gen5SpirvStage.Compute || request.EnableGraphicsSubgroupOperations; + _nativeHalfConversionExact = request.NativeHalfConversionExact; + _zeroOutOfBoundsBufferReads = request.ZeroOutOfBoundsBufferReads; _waveLaneCount = request.WaveSize == 64 ? 64u : 32u; _localSizeX = Math.Max(request.LocalSizeX, 1); _localSizeY = Math.Max(request.LocalSizeY, 1); _localSizeZ = Math.Max(request.LocalSizeZ, 1); _physicalAxisOfLogical = ComputeWorkgroupAxisOrder(_localSizeX, _localSizeY, _localSizeZ); - _emulateWave64 = _stage == Gen5SpirvStage.Compute && _waveLaneCount == 64 && (ulong)_localSizeX * _localSizeY * _localSizeZ == 64; + _emulateWave64 = _stage == Gen5SpirvStage.Compute && _waveLaneCount == 64 && + ((ulong)_localSizeX * _localSizeY * _localSizeZ == 64 || request.CooperativeWave64Workgroup); _requiredVertexOutputCount = request.RequiredVertexOutputCount; _pixelInputEnable = request.PixelInputEnable; _pixelInputAddress = request.PixelInputAddress; @@ -150,29 +205,41 @@ public static partial class Gen5SpirvTranslator _storageUlongPointer = _module.TypePointer(SpirvStorageClass.StorageBuffer, _ulongType); foreach (var binding in layout.Descriptors) { - var number = BindingLayout.NativeBindingIndex(request.Stage, binding.Kind); + var isImage = ImageDescriptorBinding.ResourceClass(binding.Kind) != ImageResourceClass.None; + var number = layout.UsesBindlessImages && isImage + ? ImageDescriptorBinding.ResourceClass(binding.Kind) == ImageResourceClass.Sampled + ? BindingLayout.BindlessSampledImageBinding + : BindingLayout.BindlessStorageImageBinding + : BindingLayout.NativeBindingIndex(request.Stage, binding.Kind); switch (binding.Kind) { case DescriptorBindingKind.Buffers: - DeclareBufferArray((uint)binding.Resources.Count, number); + DeclareBufferArray((uint)binding.Resources.Count, number, request.Bindings.UsesBindlessImages ? 1u : 0u); break; case DescriptorBindingKind.Samplers: - DeclareSamplerArray((uint)binding.Resources.Count, number); + DeclareSamplerArray((uint)binding.Resources.Count, number, request.Bindings.UsesBindlessImages ? 1u : 0u); break; case DescriptorBindingKind.GlobalDataShare: - _globalDataShare = DeclareWordBlock("globalDataShare", number); + _globalDataShare = DeclareWordBlock("globalDataShare", number, request.Bindings.UsesBindlessImages ? 1u : 0u); break; case DescriptorBindingKind.DeviceAddressPageTable: - _pageTable = DeclareDeviceAddressPageTable(number); + _pageTable = DeclareDeviceAddressPageTable(number, request.Bindings.UsesBindlessImages ? 1u : 0u); break; case DescriptorBindingKind.FaultBuffer: - _faultBuffer = DeclareWordBlock("faultBuffer", number); + _faultBuffer = DeclareWordBlock("faultBuffer", number, request.Bindings.UsesBindlessImages ? 1u : 0u); break; case DescriptorBindingKind.FlattenedResourceTable: - _flattenedTable = DeclareWordBlock("flattenedResourceTable", number); + _flattenedTable = DeclareWordBlock("flattenedResourceTable", number, request.Bindings.UsesBindlessImages ? 1u : 0u); break; case DescriptorBindingKind.ShaderData: - _shaderData = DeclareWordBlock("shaderData", number); + _shaderData = DeclareWordBlock("shaderData", number, request.Bindings.UsesBindlessImages ? 1u : 0u); + _module.AddDecoration(_shaderData, SpirvDecoration.NonWritable); + break; + case DescriptorBindingKind.RuntimeDescriptorTable: + _runtimeDescriptorTable = DeclareWordBlock("runtimeDescriptorTable", number, 1u); + break; + case DescriptorBindingKind.RuntimeDescriptorMisses: + _runtimeDescriptorMisses = DeclareWordBlock("runtimeDescriptorMisses", number, 1u); break; default: DeclareImageClass(binding, number); @@ -182,8 +249,7 @@ public static partial class Gen5SpirvTranslator if (layout.UsesPushData) { - var arrayType = _module.TypeArray(_uintType, PushData.DwordCount); - _module.AddDecoration(arrayType, SpirvDecoration.ArrayStride, sizeof(uint)); + var arrayType = _module.TypeArray(_uintType, PushData.DwordCount, sizeof(uint)); var block = _module.TypeStruct(arrayType); _module.AddDecoration(block, SpirvDecoration.Block); _module.AddMemberDecoration(block, 0, SpirvDecoration.Offset, 0); @@ -208,20 +274,53 @@ public static partial class Gen5SpirvTranslator _module.AddCapability(SpirvCapability.PhysicalStorageBufferAddresses); _module.SetPhysicalStorageBuffer64MemoryModel(); _physicalUintPointer = _module.TypePointer(SpirvStorageClass.PhysicalStorageBuffer, _uintType); - var privateUlongPointer = _module.TypePointer(SpirvStorageClass.Private, _ulongType); - _deviceEntryScratch = _module.AddGlobalVariable(privateUlongPointer, SpirvStorageClass.Private, _module.Constant64(_ulongType, 0)); _deviceWordScratch = _module.AddGlobalVariable(_privateUintPointer, SpirvStorageClass.Private, UInt(0)); - _module.AddName(_deviceEntryScratch, "deviceAddressEntry"); _module.AddName(_deviceWordScratch, "deviceAddressWord"); - _interfaces.Add(_deviceEntryScratch); _interfaces.Add(_deviceWordScratch); - if (request.Memory.Entries.Any(static memory => memory.DeviceDescriptor)) + _pendingFaultWord = _module.AddGlobalVariable(_privateUintPointer, SpirvStorageClass.Private, UInt(0)); + _pendingFaultBits = _module.AddGlobalVariable(_privateUintPointer, SpirvStorageClass.Private, UInt(0)); + _module.AddName(_pendingFaultWord, "pendingFaultWord"); + _module.AddName(_pendingFaultBits, "pendingFaultBits"); + _interfaces.Add(_pendingFaultWord); + _interfaces.Add(_pendingFaultBits); + _pendingWrittenWord = _module.AddGlobalVariable(_privateUintPointer, SpirvStorageClass.Private, UInt(0)); + _pendingWrittenBits = _module.AddGlobalVariable(_privateUintPointer, SpirvStorageClass.Private, UInt(0)); + _module.AddName(_pendingWrittenWord, "pendingWrittenWord"); + _module.AddName(_pendingWrittenBits, "pendingWrittenBits"); + _interfaces.Add(_pendingWrittenWord); + _interfaces.Add(_pendingWrittenBits); + for (var index = 0; index < _deviceStoreValueScratch.Length; index++) + { + _deviceStoreValueScratch[index] = _module.AddGlobalVariable(_privateUintPointer, SpirvStorageClass.Private, UInt(0)); + _deviceStoreMaskScratch[index] = _module.AddGlobalVariable(_privateUintPointer, SpirvStorageClass.Private, UInt(0)); + _module.AddName(_deviceStoreValueScratch[index], $"deviceStoreValue{index}"); + _module.AddName(_deviceStoreMaskScratch[index], $"deviceStoreMask{index}"); + _interfaces.Add(_deviceStoreValueScratch[index]); + _interfaces.Add(_deviceStoreMaskScratch[index]); + } + + _deviceStoreBytesScratch = _module.AddGlobalVariable(_privateUintPointer, SpirvStorageClass.Private, UInt(0)); + _module.AddName(_deviceStoreBytesScratch, "deviceStoreBytes"); + _interfaces.Add(_deviceStoreBytesScratch); + if (request.Memory.Entries.Any(static memory => memory.DeviceDescriptor) || + request.Program.Instructions.Any(instruction => IsFormatBufferLoad(instruction.Opcode))) { _deviceBufferWordScratch = _module.AddGlobalVariable(_privateUintPointer, SpirvStorageClass.Private, UInt(0)); _module.AddName(_deviceBufferWordScratch, "deviceBufferWord"); _interfaces.Add(_deviceBufferWordScratch); } } + else if (layout.UsesTessellationData) + { + _module.AddCapability(SpirvCapability.PhysicalStorageBufferAddresses); + _module.SetPhysicalStorageBuffer64MemoryModel(); + _physicalUintPointer = _module.TypePointer(SpirvStorageClass.PhysicalStorageBuffer, _uintType); + } + if (request.TessellationHull is not null) + { + _tessellationVertexIndexScratch = _module.AddGlobalVariable(_privateUintPointer, SpirvStorageClass.Private, UInt(0)); + _interfaces.Add(_tessellationVertexIndexScratch); + } foreach (var memoryIndex in request.IndirectKeyMemoryIndices) { @@ -255,7 +354,7 @@ public static partial class Gen5SpirvTranslator return _addressRuntimeArray; } - private void DeclareBufferArray(uint count, uint bindingNumber) + private void DeclareBufferArray(uint count, uint bindingNumber, uint descriptorSet) { var runtimeArray = WordRuntimeArray(); var block = _module.TypeStruct(runtimeArray); @@ -266,26 +365,26 @@ public static partial class Gen5SpirvTranslator _storageBlockPointer = _module.TypePointer(SpirvStorageClass.StorageBuffer, block); _globalBuffers = _module.AddGlobalVariable(descriptorsPointer, SpirvStorageClass.StorageBuffer); _module.AddName(_globalBuffers, "guestBuffers"); - _module.AddDecoration(_globalBuffers, SpirvDecoration.DescriptorSet, 0); + _module.AddDecoration(_globalBuffers, SpirvDecoration.DescriptorSet, descriptorSet); _module.AddDecoration(_globalBuffers, SpirvDecoration.Binding, bindingNumber); _interfaces.Add(_globalBuffers); } // One storage block holding a runtime array of dwords. - private uint DeclareWordBlock(string name, uint bindingNumber) + private uint DeclareWordBlock(string name, uint bindingNumber, uint descriptorSet) { var block = _module.TypeStruct(WordRuntimeArray()); _module.AddDecoration(block, SpirvDecoration.Block); _module.AddMemberDecoration(block, 0, SpirvDecoration.Offset, 0); var variable = _module.AddGlobalVariable(_module.TypePointer(SpirvStorageClass.StorageBuffer, block), SpirvStorageClass.StorageBuffer); _module.AddName(variable, name); - _module.AddDecoration(variable, SpirvDecoration.DescriptorSet, 0); + _module.AddDecoration(variable, SpirvDecoration.DescriptorSet, descriptorSet); _module.AddDecoration(variable, SpirvDecoration.Binding, bindingNumber); _interfaces.Add(variable); return variable; } - private uint DeclareDeviceAddressPageTable(uint bindingNumber) + private uint DeclareDeviceAddressPageTable(uint bindingNumber, uint descriptorSet) { var block = _module.TypeStruct(AddressRuntimeArray()); _module.AddDecoration(block, SpirvDecoration.Block); @@ -293,20 +392,20 @@ public static partial class Gen5SpirvTranslator _module.AddMemberDecoration(block, 0, SpirvDecoration.NonWritable); var variable = _module.AddGlobalVariable(_module.TypePointer(SpirvStorageClass.StorageBuffer, block), SpirvStorageClass.StorageBuffer); _module.AddName(variable, "deviceAddressPageTable"); - _module.AddDecoration(variable, SpirvDecoration.DescriptorSet, 0); + _module.AddDecoration(variable, SpirvDecoration.DescriptorSet, descriptorSet); _module.AddDecoration(variable, SpirvDecoration.Binding, bindingNumber); _interfaces.Add(variable); return variable; } - private void DeclareSamplerArray(uint count, uint bindingNumber) + private void DeclareSamplerArray(uint count, uint bindingNumber, uint descriptorSet) { _samplerType = _module.TypeSampler(); var arrayType = _module.TypeArray(_samplerType, count); _samplerPointer = _module.TypePointer(SpirvStorageClass.UniformConstant, _samplerType); _samplerArray = _module.AddGlobalVariable(_module.TypePointer(SpirvStorageClass.UniformConstant, arrayType), SpirvStorageClass.UniformConstant); _module.AddName(_samplerArray, "samplers"); - _module.AddDecoration(_samplerArray, SpirvDecoration.DescriptorSet, 0); + _module.AddDecoration(_samplerArray, SpirvDecoration.DescriptorSet, descriptorSet); _module.AddDecoration(_samplerArray, SpirvDecoration.Binding, bindingNumber); _interfaces.Add(_samplerArray); } @@ -366,14 +465,22 @@ public static partial class Gen5SpirvTranslator } var imageType = _module.TypeImage(componentType, spirvDimension, depth: false, arrayed, multisampled, sampled: isStorage ? 2u : 1u, format); - var arrayType = _module.TypeArray(imageType, (uint)binding.Resources.Count); + var slotOffset = _imageSlotTableDwordCount; + _imageSlotTableDwordCount += (uint)binding.Resources.Count; + var arrayType = _request.Bindings.UsesBindlessImages + ? _module.TypeRuntimeArray(imageType) + : _module.TypeArray(imageType, (uint)binding.Resources.Count); + if (_request.Bindings.UsesBindlessImages) + { + _module.AddCapability(SpirvCapability.RuntimeDescriptorArray); + } var elementPointer = _module.TypePointer(SpirvStorageClass.UniformConstant, imageType); var variable = _module.AddGlobalVariable(_module.TypePointer(SpirvStorageClass.UniformConstant, arrayType), SpirvStorageClass.UniformConstant); _module.AddName(variable, $"{binding.Kind}"); _module.AddDecoration(variable, SpirvDecoration.DescriptorSet, 0); _module.AddDecoration(variable, SpirvDecoration.Binding, bindingNumber); _interfaces.Add(variable); - _imageClasses[binding.Kind] = new LayoutImageClass(variable, imageType, elementPointer, componentType, kind, isStorage, arrayed, cube, multisampled, spirvDimension, binding.Resources); + _imageClasses[binding.Kind] = new LayoutImageClass(variable, imageType, elementPointer, componentType, kind, isStorage, arrayed, cube, multisampled, spirvDimension, binding.Resources, slotOffset); } // ---- initial state ---- @@ -413,9 +520,339 @@ public static partial class Gen5SpirvTranslator return LoadBlockWord(_shaderData, dwordIndex); } + // ---- runtime descriptors ---- + + // An access whose image descriptor has no plan-time source looks its words up in the + // host's table (RuntimeDescriptorTable). Sampling accesses retain the historical float + // view key. Image loads derive their numeric view from the descriptor format, then fetch + // through the matching bindless image array. A missing descriptor reads slot 0 and is + // recorded for the host. + private bool TryEmitRuntimeDescriptorImage(Gen5ShaderInstruction instruction, Gen5ImageControl image, MemoryAccessInfo entry, out string error) + { + if (_runtimeDescriptorTable == 0 || _runtimeDescriptorMisses == 0) + { + error = "runtime image descriptors without their table bindings"; + return false; + } + + var imageKey = new uint[1 + RuntimeDescriptorTable.ImageWordCount]; + for (uint word = 0; word < RuntimeDescriptorTable.ImageWordCount; word++) + { + // A 128-bit descriptor has four words; the rest of the key is zero. + imageKey[1 + word] = entry.ImageR128 && word >= 4 ? UInt(0) : LoadS(image.ScalarResource + word); + } + + if (!entry.NeedsSampler) + { + // The PS5 descriptor controls the numeric interpretation of an image load. + // Keep its format-driven class in the key so an R32_UINT and an R32_SINT view + // of the same image cannot alias each other in the table. + var numericClass = RuntimeImageNumericClass(imageKey[2]); + imageKey[0] = BitwiseOr( + UInt(RuntimeDescriptorTable.ViewClass(entry.ImageDimension)), + ShiftLeftLogical(numericClass, UInt(RuntimeDescriptorTable.ViewClassNumericShift))); + var runtimeImageSlot = LookupRuntimeDescriptor(imageKey, image: true); + _module.AddCapability(SpirvCapability.ShaderNonUniform); + _module.AddCapability(SpirvCapability.SampledImageArrayNonUniformIndexing); + _module.AddDecoration(runtimeImageSlot, SpirvDecoration.NonUniform); + + string? branchError = null; + // A descriptor array element has one numeric view. Fetch only from the array + // matching that view rather than issuing invalid loads through the other typed + // declarations. Unknown formats use float and remain absent from the host table. + EmitConditional(_module.AddInstruction(SpirvOp.IEqual, _boolType, numericClass, UInt(0)), () => + branchError ??= EmitRuntimeDescriptorFetch(instruction, image, entry, runtimeImageSlot, ImageNumericClass.Float)); + EmitConditional(_module.AddInstruction(SpirvOp.IEqual, _boolType, numericClass, UInt(1)), () => + branchError ??= EmitRuntimeDescriptorFetch(instruction, image, entry, runtimeImageSlot, ImageNumericClass.Uint)); + EmitConditional(_module.AddInstruction(SpirvOp.IEqual, _boolType, numericClass, UInt(2)), () => + branchError ??= EmitRuntimeDescriptorFetch(instruction, image, entry, runtimeImageSlot, ImageNumericClass.Sint)); + error = branchError ?? string.Empty; + return branchError is null; + } + + var view = RuntimeImageArray(entry.ImageDimension, ImageNumericClass.Float); + imageKey[0] = UInt(RuntimeDescriptorTable.ViewClass(entry.ImageDimension)); + var samplerKey = new uint[RuntimeDescriptorTable.SamplerWordCount]; + for (uint word = 0; word < samplerKey.Length; word++) + { + samplerKey[word] = LoadS(image.ScalarSampler + word); + } + + if ((entry.ImageSampleFlags & ImageSampleFlags.Adjust) != 0) + { + samplerKey[3] = BitwiseAnd(samplerKey[3], UInt(~DescriptorConstants.SamplerDword3ReservedMask)); + } + var imageSlot = LookupRuntimeDescriptor(imageKey, image: true); + var samplerSlot = LookupRuntimeDescriptor(samplerKey, image: false); + + _module.AddCapability(SpirvCapability.ShaderNonUniform); + _module.AddCapability(SpirvCapability.SampledImageArrayNonUniformIndexing); + var imagePointer = _module.AddInstruction(SpirvOp.AccessChain, view.ElementPointer, view.Variable, imageSlot); + var imageValue = Load(view.ImageType, imagePointer); + var samplerArray = RuntimeSamplerArray(); + var samplerPointer = _module.AddInstruction(SpirvOp.AccessChain, _runtimeSamplerPointer, samplerArray, samplerSlot); + var sampler = Load(_samplerType, samplerPointer); + var objectType = _module.TypeSampledImage(view.ImageType); + var imageObject = _module.AddInstruction(SpirvOp.SampledImage, objectType, imageValue, sampler); + foreach (var value in new[] { imageSlot, imagePointer, imageValue, samplerSlot, samplerPointer, sampler, imageObject }) + { + _module.AddDecoration(value, SpirvDecoration.NonUniform); + } + + // The host's view applies the descriptor's swizzle and format. + var resource = new SpirvImageResource( + imagePointer, + view.ImageType, + objectType, + view.ComponentType, + view.VectorType, + view.Kind, + IsStorage: false, + view.Arrayed, + Dimension: view.Dimension, + ShaderSwizzle: DescriptorConstants.IdentityImageSwizzle); + return EmitImageOperation(instruction, image, resource, imageObject, DescriptorConstants.IdentityImageSwizzle, UInt(0), out error); + } + + private string? EmitRuntimeDescriptorFetch(Gen5ShaderInstruction instruction, Gen5ImageControl image, + MemoryAccessInfo entry, uint imageSlot, ImageNumericClass numericClass) + { + var view = RuntimeImageArray(entry.ImageDimension, numericClass); + var imagePointer = _module.AddInstruction(SpirvOp.AccessChain, view.ElementPointer, view.Variable, imageSlot); + var imageValue = Load(view.ImageType, imagePointer); + _module.AddDecoration(imagePointer, SpirvDecoration.NonUniform); + _module.AddDecoration(imageValue, SpirvDecoration.NonUniform); + var resource = new SpirvImageResource( + imagePointer, + view.ImageType, + view.ImageType, + view.ComponentType, + view.VectorType, + view.Kind, + IsStorage: false, + view.Arrayed, + Dimension: view.Dimension, + ShaderSwizzle: DescriptorConstants.IdentityImageSwizzle); + var mipLevel = instruction.Opcode == "ImageLoadMip" + ? LoadImageIntegerAddress(image, (int)ImageCoordinateComponentCount(resource)) + : UInt(0); + return EmitImageOperation(instruction, image, resource, imageValue, + DescriptorConstants.IdentityImageSwizzle, mipLevel, out var error) ? null : error; + } + + // The heap's sampled images of one dimension and numeric view, declared once per module. + private (uint Variable, uint ImageType, uint ElementPointer, uint ComponentType, uint VectorType, + ImageComponentKind Kind, SpirvImageDim Dimension, bool Arrayed) RuntimeImageArray( + ImageDimension dimension, ImageNumericClass numericClass) + { + var key = (dimension, numericClass); + if (_runtimeImageArrays.TryGetValue(key, out var declared)) + { + return declared; + } + + var spirvDimension = dimension switch + { + ImageDimension.Dim1D or ImageDimension.Dim1DArray => SpirvImageDim.Dim1D, + ImageDimension.Dim3D => SpirvImageDim.Dim3D, + _ => SpirvImageDim.Dim2D, + }; + if (spirvDimension == SpirvImageDim.Dim1D) + { + _module.AddCapability(SpirvCapability.Sampled1D); + } + + var arrayed = dimension is ImageDimension.Dim1DArray or ImageDimension.Dim2DArray; + var kind = numericClass switch + { + ImageNumericClass.Uint => ImageComponentKind.Uint, + ImageNumericClass.Sint => ImageComponentKind.Sint, + _ => ImageComponentKind.Float, + }; + var componentType = kind switch + { + ImageComponentKind.Uint => _uintType, + ImageComponentKind.Sint => _intType, + _ => _floatType, + }; + var vectorType = _module.TypeVector(componentType, 4); + var imageType = _module.TypeImage(componentType, spirvDimension, depth: false, arrayed, multisampled: false, sampled: 1u, SpirvImageFormat.Unknown); + _module.AddCapability(SpirvCapability.RuntimeDescriptorArray); + var arrayType = _module.TypeRuntimeArray(imageType); + var elementPointer = _module.TypePointer(SpirvStorageClass.UniformConstant, imageType); + var variable = _module.AddGlobalVariable(_module.TypePointer(SpirvStorageClass.UniformConstant, arrayType), SpirvStorageClass.UniformConstant); + _module.AddName(variable, $"runtimeImages{dimension}{numericClass}"); + _module.AddDecoration(variable, SpirvDecoration.DescriptorSet, 0); + _module.AddDecoration(variable, SpirvDecoration.Binding, BindingLayout.BindlessSampledImageBinding); + _interfaces.Add(variable); + declared = (variable, imageType, elementPointer, componentType, vectorType, kind, spirvDimension, arrayed); + _runtimeImageArrays.Add(key, declared); + return declared; + } + + // Runtime descriptors are intentionally not specialized to the format observed while a + // pipeline is compiled. Use the same format classification as the host's descriptor + // materializer, represented compactly as float=0, uint=1 and sint=2. + private uint RuntimeImageNumericClass(uint descriptorWord1) + { + if (_runtimeImageNumericClassTable == 0) + { + var entries = new uint[GuestImageFormat.MaxFormat + 1]; + for (uint tableFormat = 0; tableFormat < entries.Length; tableFormat++) + { + entries[tableFormat] = RuntimeDescriptorTable.NumericClassCode(GuestImageFormat.SampledNumericClass(tableFormat)); + } + + var type = _module.TypeArray(_uintType, (uint)entries.Length); + _runtimeImageNumericClassTable = _module.AddGlobalVariable( + _module.TypePointer(SpirvStorageClass.Private, type), SpirvStorageClass.Private, + _module.ConstantComposite(type, entries.Select(UInt).ToArray())); + _module.AddName(_runtimeImageNumericClassTable, "runtimeImageNumericClasses"); + _interfaces.Add(_runtimeImageNumericClassTable); + } + + var descriptorFormat = BitwiseAnd(ShiftRightLogical(descriptorWord1, UInt(20)), UInt(0x1ff)); + var inRange = _module.AddInstruction(SpirvOp.ULessThan, _boolType, descriptorFormat, UInt(GuestImageFormat.MaxFormat + 1)); + var safeFormat = _module.AddInstruction(SpirvOp.Select, _uintType, inRange, descriptorFormat, UInt(0)); + return Load(_uintType, _module.AddInstruction(SpirvOp.AccessChain, _privateUintPointer, + _runtimeImageNumericClassTable, safeFormat)); + } + + private uint RuntimeSamplerArray() + { + if (_runtimeSamplerArray != 0) + { + return _runtimeSamplerArray; + } + + _samplerType = _module.TypeSampler(); + _runtimeSamplerPointer = _module.TypePointer(SpirvStorageClass.UniformConstant, _samplerType); + _module.AddCapability(SpirvCapability.RuntimeDescriptorArray); + var arrayType = _module.TypeRuntimeArray(_samplerType); + _runtimeSamplerArray = _module.AddGlobalVariable(_module.TypePointer(SpirvStorageClass.UniformConstant, arrayType), SpirvStorageClass.UniformConstant); + _module.AddName(_runtimeSamplerArray, "runtimeSamplers"); + _module.AddDecoration(_runtimeSamplerArray, SpirvDecoration.DescriptorSet, 0); + _module.AddDecoration(_runtimeSamplerArray, SpirvDecoration.Binding, BindingLayout.BindlessSamplerBinding); + _interfaces.Add(_runtimeSamplerArray); + return _runtimeSamplerArray; + } + + // One shared lookup function per table (image, sampler), emitted after the program. + private readonly uint[] _runtimeLookupFunctions = new uint[2]; + + // The heap slot of a descriptor key. Every access calls one shared function: inlined, the + // hash and probes of each lookup made a shader with many runtime descriptor accesses + // several times larger and its pipeline many times slower to compile. + private uint LookupRuntimeDescriptor(uint[] key, bool image) + { + var kind = image ? 0 : 1; + if (_runtimeLookupFunctions[kind] == 0) + { + _runtimeLookupFunctions[kind] = _module.AllocateId(); + } + + var active = _module.AddInstruction(SpirvOp.Select, _uintType, Load(_boolType, _exec), UInt(1), UInt(0)); + return _module.AddInstruction(SpirvOp.FunctionCall, _uintType, [_runtimeLookupFunctions[kind], .. key, active]); + } + + private void EmitRuntimeLookupFunctions() + { + for (var kind = 0; kind < _runtimeLookupFunctions.Length; kind++) + { + var function = _runtimeLookupFunctions[kind]; + if (function == 0) + { + continue; + } + + var image = kind == 0; + var keyWords = image ? 1 + (int)RuntimeDescriptorTable.ImageWordCount : (int)RuntimeDescriptorTable.SamplerWordCount; + var type = _module.TypeFunction(_uintType, [.. Enumerable.Repeat(_uintType, keyWords + 1)]); + _module.BeginFunction(_uintType, type, function, SpirvFunctionControl.DontInline); + _module.AddName(function, image ? "lookupRuntimeImage" : "lookupRuntimeSampler"); + var key = new uint[keyWords]; + for (var word = 0; word < keyWords; word++) + { + key[word] = _module.AddFunctionParameter(_uintType); + } + + var active = _module.AddFunctionParameter(_uintType); + _module.AddLabel(); + _module.AddStatement(SpirvOp.ReturnValue, EmitRuntimeDescriptorProbe(key, image, + _module.AddInstruction(SpirvOp.INotEqual, _boolType, active, UInt(0)))); + _module.EndFunction(); + } + } + + // A bounded linear probe of the host table; slot 0 and a recorded miss when the table + // does not hold the key. + private uint EmitRuntimeDescriptorProbe(uint[] key, bool image, uint active) + { + var table = _runtimeDescriptorTable; + var capacity = LoadBlockWord(table, UInt(image ? RuntimeDescriptorTable.ImageCapacityDword : RuntimeDescriptorTable.SamplerCapacityDword)); + var entries = LoadBlockWord(table, UInt(image ? RuntimeDescriptorTable.ImageEntriesDword : RuntimeDescriptorTable.SamplerEntriesDword)); + var entryDwords = image ? RuntimeDescriptorTable.ImageEntryDwords : RuntimeDescriptorTable.SamplerEntryDwords; + var mask = _module.AddInstruction(SpirvOp.ISub, _uintType, capacity, UInt(1)); + var hash = UInt(RuntimeDescriptorTable.HashSeed); + foreach (var word in key) + { + hash = _module.AddInstruction(SpirvOp.IMul, _uintType, + _module.AddInstruction(SpirvOp.BitwiseXor, _uintType, hash, word), UInt(RuntimeDescriptorTable.HashPrime)); + } + + var empty = _module.AddInstruction(SpirvOp.IEqual, _boolType, capacity, UInt(0)); + var slotPlusOne = UInt(0); + for (uint probe = 0; probe < RuntimeDescriptorTable.ProbeCount; probe++) + { + var index = BitwiseAnd(IAdd(hash, UInt(probe)), mask); + var entry = IAdd(entries, _module.AddInstruction(SpirvOp.IMul, _uintType, index, UInt(entryDwords))); + var stored = LoadBlockWord(table, entry); + var match = LogicalAnd(LogicalNot(empty), _module.AddInstruction(SpirvOp.INotEqual, _boolType, stored, UInt(0))); + for (var word = 0; word < key.Length; word++) + { + var tableWord = LoadBlockWord(table, IAdd(entry, UInt((uint)word + 1))); + match = LogicalAnd(match, _module.AddInstruction(SpirvOp.IEqual, _boolType, tableWord, key[word])); + } + + var take = LogicalAnd(match, _module.AddInstruction(SpirvOp.IEqual, _boolType, slotPlusOne, UInt(0))); + slotPlusOne = _module.AddInstruction(SpirvOp.Select, _uintType, take, stored, slotPlusOne); + } + + var missing = _module.AddInstruction(SpirvOp.IEqual, _boolType, slotPlusOne, UInt(0)); + // Only an active lane reads its descriptor; an inactive one may hold anything. + EmitConditional(_module.AddInstruction(SpirvOp.LogicalAnd, _boolType, missing, active), + () => RecordRuntimeDescriptorMiss(key, image)); + return _module.AddInstruction(SpirvOp.Select, _uintType, missing, UInt(0), + _module.AddInstruction(SpirvOp.ISub, _uintType, slotPlusOne, UInt(1))); + } + + private void RecordRuntimeDescriptorMiss(uint[] key, bool image) + { + var misses = _runtimeDescriptorMisses; + var counter = BlockWordPointer(misses, UInt(image ? RuntimeDescriptorTable.ImageMissCountDword : RuntimeDescriptorTable.SamplerMissCountDword)); + var record = _module.AddInstruction(SpirvOp.AtomicIAdd, _uintType, counter, UInt(1), UInt(0), UInt(1)); + var capacity = image ? RuntimeDescriptorTable.ImageMissCapacity : RuntimeDescriptorTable.SamplerMissCapacity; + var recordDwords = image ? RuntimeDescriptorTable.ImageMissRecordDwords : RuntimeDescriptorTable.SamplerMissRecordDwords; + var offset = image ? RuntimeDescriptorTable.ImageMissOffset : RuntimeDescriptorTable.SamplerMissOffset; + EmitConditional(_module.AddInstruction(SpirvOp.ULessThan, _boolType, record, UInt(capacity)), () => + { + var first = IAdd(UInt(offset), _module.AddInstruction(SpirvOp.IMul, _uintType, record, UInt(recordDwords))); + for (var word = 0; word < key.Length; word++) + { + var dword = IAdd(first, UInt((uint)word)); + EmitConditional(IsBlockWordInRange(misses, dword), () => Store(BlockWordPointer(misses, dword), key[word])); + } + }); + } + // A bounds-checked dword of a storage block; outside the block reads zero. private uint LoadBlockWord(uint block, uint dwordIndex) { + if (_zeroOutOfBoundsBufferReads) + { + return Load(_uintType, BlockWordPointer(block, dwordIndex)); + } + var length = _module.AddInstruction(SpirvOp.ArrayLength, _uintType, block, 0); var inRange = _module.AddInstruction(SpirvOp.ULessThan, _boolType, dwordIndex, length); var safeIndex = _module.AddInstruction(SpirvOp.Select, _uintType, inRange, dwordIndex, UInt(0)); @@ -434,15 +871,39 @@ public static partial class Gen5SpirvTranslator dwordIndex, _module.AddInstruction(SpirvOp.ArrayLength, _uintType, block, 0)); - private uint LoadFlattenedWord(uint slot) => LoadBlockWord(_flattenedTable, slot); + private uint LoadFlattenedWord(uint slot) + { + if (_request.Bindings.UsesBindlessImages) + { + slot = IAdd(UInt(BindingLayout.ImageSlotTableDwordCount(_request.Resources.Info)), slot); + } + + return LoadBlockWord(_flattenedTable, slot); + } + + private uint LoadImageSlot(LayoutImageClass imageClass, uint element) + { + var slot = IAdd(UInt(imageClass.SlotOffset), element); + return LoadBlockWord(_flattenedTable, slot); + } // A planned slot is inside the table the host binds, so its load skips the runtime bounds // check. Large shaders make hundreds of these reads, and every check costs the Metal // compiler time (one large pixel shader made about 376 of them). - private uint LoadFlattenedSlot(uint slot) => - slot < _request.FlattenedTableReservedWords - ? Load(_uintType, BlockWordPointer(_flattenedTable, UInt(slot))) - : LoadFlattenedWord(UInt(slot)); + private uint LoadFlattenedSlot(uint slot) + { + if (slot >= _request.FlattenedTableReservedWords) + { + return LoadFlattenedWord(UInt(slot)); + } + + // Reserved slots address guest words, after the bindless image indices + // that the host prepends to the same buffer. + var prefix = _request.Bindings.UsesBindlessImages + ? BindingLayout.ImageSlotTableDwordCount(_request.Resources.Info) + : 0; + return Load(_uintType, BlockWordPointer(_flattenedTable, UInt(prefix + slot))); + } // The shader base the host pushes for this draw, as two dwords. private (uint Low, uint High) LoadShaderBase() @@ -507,6 +968,7 @@ public static partial class Gen5SpirvTranslator // Resolves a guest address through the page table. A missing page records a // fault and resolves to an invalid pointer; an address past the table reads as unmapped. private uint _deviceAddressInstructionPc; + private uint? _deviceAddressInstructionPcValue; private (uint Pointer, uint Valid) ResolveDeviceAddress(uint address64) { @@ -515,17 +977,17 @@ public static partial class Gen5SpirvTranslator var tableLength = _module.AddInstruction(SpirvOp.ArrayLength, _uintType, _pageTable, 0); var inTable = ULessThan64(pageIndex64, Widen(tableLength)); var pageIndex = Narrow(pageIndex64); - Store(_deviceEntryScratch, ULong(0)); - EmitConditional(inTable, () => - { - var entryPointer = _module.AddInstruction(SpirvOp.AccessChain, _storageUlongPointer, _pageTable, UInt(0), pageIndex); - Store(_deviceEntryScratch, Load(_ulongType, entryPointer)); - }); - var entry = Load(_ulongType, _deviceEntryScratch); + // Read without a branch: an index outside the table reads entry 0 (the host always + // binds CachingPageCount entries) and the select discards it. A branch and a scratch + // variable per page walk cost the driver seconds on shaders with many device accesses. + var safeIndex = _module.AddInstruction(SpirvOp.Select, _uintType, inTable, pageIndex, UInt(0)); + var entryPointer = _module.AddInstruction(SpirvOp.AccessChain, _storageUlongPointer, _pageTable, UInt(0), safeIndex); + var entry = _module.AddInstruction(SpirvOp.Select, _ulongType, inTable, Load(_ulongType, entryPointer), ULong(0)); var mapped = _module.AddInstruction(SpirvOp.INotEqual, _boolType, entry, ULong(0)); - EmitConditional(LogicalAnd(inTable, LogicalNot(mapped)), () => + var missing = LogicalAnd(inTable, LogicalNot(mapped)); + if (_request.TraceDeviceAddressFaults) { - if (_request.TraceDeviceAddressFaults) + EmitConditional(missing, () => { var length = _module.AddInstruction(SpirvOp.ArrayLength, _uintType, _faultBuffer, 0); var recordStart = _module.AddInstruction(SpirvOp.ISub, _uintType, length, UInt(8)); @@ -534,22 +996,81 @@ public static partial class Gen5SpirvTranslator EmitConditional(_module.AddInstruction(SpirvOp.IEqual, _boolType, claim, UInt(0)), () => { uint[] values = [UInt((uint)_request.Hash), UInt((uint)(_request.Hash >> 32)), - UInt(_deviceAddressInstructionPc), Narrow(address64), + _deviceAddressInstructionPcValue ?? UInt(_deviceAddressInstructionPc), Narrow(address64), Narrow(_module.AddInstruction(SpirvOp.ShiftRightLogical, _ulongType, address64, ULong(32))), UInt((uint)_request.Stage), UInt(0)]; for (var index = 0; index < values.Length; index++) Store(BlockWordPointer(_faultBuffer, IAdd(recordStart, UInt((uint)index + 1))), values[index]); }); - } - var word = ShiftRightLogical(pageIndex, UInt(5)); - var bit = ShiftLeftLogical(UInt(1), BitwiseAnd(pageIndex, UInt(31))); - EmitConditional(IsBlockWordInRange(_faultBuffer, word), () => - _module.AddInstruction(SpirvOp.AtomicOr, _uintType, BlockWordPointer(_faultBuffer, word), UInt(1), UInt(0), bit)); - }); + }); + } + + // Merged by selects, not under a branch: a branch per page walk costs the driver + // nearly as much as the atomic it replaces. + var word = ShiftRightLogical(pageIndex, UInt(5)); + var bit = ShiftLeftLogical(UInt(1), BitwiseAnd(pageIndex, UInt(31))); + var pendingWord = Load(_uintType, _pendingFaultWord); + var pendingBits = Load(_uintType, _pendingFaultBits); + var merge = LogicalAnd(missing, _module.AddInstruction(SpirvOp.LogicalOr, _boolType, + _module.AddInstruction(SpirvOp.IEqual, _boolType, pendingBits, UInt(0)), + _module.AddInstruction(SpirvOp.IEqual, _boolType, pendingWord, word))); + Store(_pendingFaultWord, _module.AddInstruction(SpirvOp.Select, _uintType, merge, word, pendingWord)); + Store(_pendingFaultBits, _module.AddInstruction(SpirvOp.Select, _uintType, merge, BitwiseOr(pendingBits, bit), pendingBits)); var pointer = IAdd64(entry, And64(masked, ULong(DeviceAddressPageSize - 1))); return (pointer, mapped); } + // Reports the page of a store or atomic through a device address. The written-page bitmap + // follows the fault bitmap in the fault buffer; the host marks those pages GPU-owned, as it + // does for written bindings. An invocation merges pages of one bitmap word and publishes the + // word when it moves to another; a pixel invocation, which a discard can end, publishes at once. + private void NoteDeviceWrite(uint address64) + { + var masked = And64(address64, ULong(DeviceAddressMask)); + var pageIndex = Narrow(_module.AddInstruction(SpirvOp.ShiftRightLogical, _ulongType, masked, ULong(DeviceAddressPageBits))); + var pageCount = _module.AddInstruction(SpirvOp.ArrayLength, _uintType, _pageTable, 0); + var word = IAdd(ShiftRightLogical(pageCount, UInt(5)), ShiftRightLogical(pageIndex, UInt(5))); + var bit = ShiftLeftLogical(UInt(1), BitwiseAnd(pageIndex, UInt(31))); + if (_stage == Gen5SpirvStage.Pixel) + { + EmitConditional(IsBlockWordInRange(_faultBuffer, word), () => + _module.AddInstruction(SpirvOp.AtomicOr, _uintType, BlockWordPointer(_faultBuffer, word), UInt(1), UInt(0), bit)); + return; + } + + var pendingWord = Load(_uintType, _pendingWrittenWord); + var pendingBits = Load(_uintType, _pendingWrittenBits); + var keep = _module.AddInstruction(SpirvOp.LogicalOr, _boolType, + _module.AddInstruction(SpirvOp.IEqual, _boolType, pendingBits, UInt(0)), + _module.AddInstruction(SpirvOp.IEqual, _boolType, pendingWord, word)); + EmitConditional(LogicalAnd(LogicalNot(keep), IsBlockWordInRange(_faultBuffer, pendingWord)), () => + _module.AddInstruction(SpirvOp.AtomicOr, _uintType, BlockWordPointer(_faultBuffer, pendingWord), UInt(1), UInt(0), pendingBits)); + Store(_pendingWrittenWord, word); + Store(_pendingWrittenBits, _module.AddInstruction(SpirvOp.Select, _uintType, keep, BitwiseOr(pendingBits, bit), bit)); + } + + // Publishes the missing pages the invocation met, once, as it ends. + private void FlushPendingDeviceFaults() + { + if (_pendingWrittenBits != 0) + { + var writtenBits = Load(_uintType, _pendingWrittenBits); + var writtenWord = Load(_uintType, _pendingWrittenWord); + EmitConditional(LogicalAnd(_module.AddInstruction(SpirvOp.INotEqual, _boolType, writtenBits, UInt(0)), IsBlockWordInRange(_faultBuffer, writtenWord)), () => + _module.AddInstruction(SpirvOp.AtomicOr, _uintType, BlockWordPointer(_faultBuffer, writtenWord), UInt(1), UInt(0), writtenBits)); + } + + if (_pendingFaultBits == 0) + { + return; + } + + var bits = Load(_uintType, _pendingFaultBits); + var word = Load(_uintType, _pendingFaultWord); + EmitConditional(LogicalAnd(_module.AddInstruction(SpirvOp.INotEqual, _boolType, bits, UInt(0)), IsBlockWordInRange(_faultBuffer, word)), () => + _module.AddInstruction(SpirvOp.AtomicOr, _uintType, BlockWordPointer(_faultBuffer, word), UInt(1), UInt(0), bits)); + } + private uint DeviceWordPointer(uint pointerValue) => _module.AddInstruction(SpirvOp.ConvertUToPtr, _physicalUintPointer, pointerValue); @@ -567,7 +1088,10 @@ public static partial class Gen5SpirvTranslator { var (pointer, valid) = ResolveDeviceAddress(address64); EmitConditional(LogicalAnd(allowed, valid), () => - _module.AddStatement(SpirvOp.Store, DeviceWordPointer(pointer), value, 2u, 4u)); + { + _module.AddStatement(SpirvOp.Store, DeviceWordPointer(pointer), value, 2u, 4u); + NoteDeviceWrite(address64); + }); } private void StoreDeviceMaskedWord(uint address64, uint value, uint mask, uint allowed) @@ -578,6 +1102,7 @@ public static partial class Gen5SpirvTranslator var (pointer, valid) = ResolveDeviceAddress(address64); EmitConditional(valid, () => { + NoteDeviceWrite(address64); var wordPointer = DeviceWordPointer(pointer); var full = _module.AddInstruction(SpirvOp.IEqual, _boolType, mask, UInt(uint.MaxValue)); EmitConditional( @@ -725,6 +1250,21 @@ public static partial class Gen5SpirvTranslator return Load(_uintType, _deviceBufferWordScratch); } + // A dword of a format element's window, read when any of its bytes is inside the buffer: + // the hardware checks the element, not its dword, and each component is bounds checked on + // its own bytes afterwards, so a byte in the last dword of a buffer whose size is not a + // multiple of four reads memory instead of zero. + private uint LoadDeviceBufferWindowWord(uint baseAddress, uint size64, uint alignedOffset) + { + var inRange = ULessThan64(Widen(alignedOffset), size64); + var address = And64( + IAdd64(baseAddress, Widen(alignedOffset)), + ULong(DeviceAddressMask & ~3ul)); + Store(_deviceBufferWordScratch, UInt(0)); + EmitConditional(inRange, () => Store(_deviceBufferWordScratch, LoadDeviceDword(address))); + return Load(_uintType, _deviceBufferWordScratch); + } + private void StoreDeviceBufferWord(uint baseAddress, uint size64, uint byteOffset, uint value) { var alignedOffset = BitwiseAnd(byteOffset, UInt(~3u)); @@ -917,6 +1457,7 @@ public static partial class Gen5SpirvTranslator var (pointer, valid) = ResolveDeviceAddress(address); EmitConditional(valid, () => { + NoteDeviceWrite(address); var original = EmitAtomic( atomicOp, _uintType, @@ -1490,7 +2031,8 @@ public static partial class Gen5SpirvTranslator out uint imageObject, out uint dstSelect, out string error, - (uint Resource, uint Element)? fixedElement = null) + (uint Resource, uint Element)? fixedElement = null, + uint dynamicElement = uint.MaxValue) { error = string.Empty; resource = default; @@ -1542,11 +2084,26 @@ public static partial class Gen5SpirvTranslator return false; } - var elementIndex = UInt(fixedElement?.Element ?? (uint)element); + var localElementIndex = dynamicElement != uint.MaxValue ? dynamicElement : UInt(fixedElement?.Element ?? (uint)element); + var elementIndex = request.Bindings.UsesBindlessImages + ? LoadImageSlot(imageClass, localElementIndex) + : localElementIndex; var elementPointer = _module.AddInstruction(SpirvOp.AccessChain, imageClass.ElementPointer, imageClass.Variable, elementIndex); var imageValue = Load(imageClass.ImageType, elementPointer); + if (dynamicElement != uint.MaxValue) + { + // The index can differ between invocations of one draw. + _module.AddCapability(SpirvCapability.ShaderNonUniform); + _module.AddCapability(imageClass.IsStorage + ? SpirvCapability.StorageImageArrayNonUniformIndexing + : SpirvCapability.SampledImageArrayNonUniformIndexing); + _module.AddDecoration(elementIndex, SpirvDecoration.NonUniform); + _module.AddDecoration(elementPointer, SpirvDecoration.NonUniform); + _module.AddDecoration(imageValue, SpirvDecoration.NonUniform); + } uint objectType; - if (UsesSampler(instruction.Opcode)) + // A multisample image is never sampled through a sampler (see EmitImageOperation). + if (UsesSampler(instruction.Opcode) && !imageClass.Multisampled) { if (_samplerArray == 0) { @@ -1569,6 +2126,8 @@ public static partial class Gen5SpirvTranslator var sampler = Load(_samplerType, samplerPointer); objectType = _module.TypeSampledImage(imageClass.ImageType); imageObject = _module.AddInstruction(SpirvOp.SampledImage, objectType, imageValue, sampler); + if (dynamicElement != uint.MaxValue) + _module.AddDecoration(imageObject, SpirvDecoration.NonUniform); } else { @@ -1590,7 +2149,8 @@ public static partial class Gen5SpirvTranslator imageClass.Dimension, imageInfo.ConversionFormat, imageInfo.ShaderSwizzle, - imageInfo.EmulatedCompareFunction); + imageInfo.EmulatedCompareFunction, + (uint)resourceIndex); dstSelect = imageInfo.ShaderSwizzle; return true; } diff --git a/src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvTranslator.StructuredControlFlow.cs b/src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvTranslator.StructuredControlFlow.cs new file mode 100644 index 0000000..2df04db --- /dev/null +++ b/src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvTranslator.StructuredControlFlow.cs @@ -0,0 +1,1113 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.ShaderCompiler; + +namespace SharpEmu.ShaderCompiler.Vulkan; + +// Structured control flow for translated programs. +// +// GCN branches are scalar: S_BRANCH and S_CBRANCH_* take the same path for every lane of a wave, +// and lane divergence lives in EXEC instead. Every invocation therefore follows the block path the +// dispatcher would run, and the block graph can be emitted as nested SPIR-V selections and loops. +// That removes the PC dispatcher, whose one loop around thousands of blocks is what makes drivers +// spend seconds compiling a large program. +// +// Supported: reducible graphs. A selection merges at the first block its arms share; returns, +// breaks and continues are legal from any depth, so they do not take part in the merge. A loop with +// several exit blocks records which exit it took in a selector, breaks, and switches on it after the +// loop. A block that arms share without merging at it is copied into each path; the copies add at +// most the size of the program, so the code never more than doubles. +// Anything else keeps the dispatcher. The walk that emits the code also runs dry first, so a +// planned program cannot fail structurally while it is emitted. +public static partial class Gen5SpirvTranslator +{ + private sealed partial class CompilationContext + { + private const int ExitNode = -1; + + // The merge of a loop with several exit blocks: a switch on the loop's selector follows it. + private const int MultiExitNode = -2; + + private static readonly bool ForceDispatcher = string.Equals( + Environment.GetEnvironmentVariable("SHARPEMU_SHADER_CONTROL_FLOW"), + "dispatcher", + StringComparison.OrdinalIgnoreCase); + + + private enum TerminatorKind + { + Exit, + Jump, + Conditional, + } + + // Taken and Fallthrough name block indices, or ExitNode when control leaves the program. + private readonly record struct BlockTerminator(TerminatorKind Kind, int Taken, int Fallthrough, string Opcode); + + private sealed class NaturalLoop + { + public required int Header; + public required HashSet Body; + public int Merge = ExitNode; + public List Exits = []; + public uint Selector; + public NaturalLoop? Parent; + } + + private sealed class ControlFlowPlan + { + public required IReadOnlyList Blocks; + public required BlockTerminator[] Terminators; + public required Dictionary LoopsByHeader; + public required NaturalLoop?[] InnermostLoop; + public required bool[] Reachable; + } + + // The kind of place a region's post-dominator walk ends at, other than a block. + private enum VirtualExit + { + None, + Return, + Continue, + Break, + Mixed, + } + + private readonly record struct MergeTarget(int Block, VirtualExit Virtual); + + private sealed class LoopFrame + { + public required NaturalLoop Loop; + public required uint ContinueLabel; + public required uint BreakLabel; + } + + private ControlFlowPlan? _structuredPlan; + private uint _structuredBody; + private bool _structuredDry; + private bool[] _structuredEmitted = []; + private int _copiedInstructions; + private int _copyBudget; + private Dictionary<(NaturalLoop? Region, int Block), MergeTarget>? _mergeCache; + + private void PrepareControlFlow(IReadOnlyList blocks) + { + var reason = "forced"; + var forced = ForceDispatcher || _request.ForceDispatcher || _request.CooperativeWave64Workgroup; + if (!forced && TryPlanStructuredControlFlow(blocks, out var plan, out reason)) + { + _structuredPlan = plan; + _structuredBody = _module.AllocateId(); + return; + } + + + } + + private bool TryEmitControlFlow(IReadOnlyList blocks, out string error) + { + error = string.Empty; + if (_request.CooperativeWave64Workgroup) return TryEmitCooperativeDispatcher(blocks, out error); + if (_structuredPlan is null) return TryEmitDispatcher(blocks, out error); + _module.AddInstruction(SpirvOp.FunctionCall, _voidType, _structuredBody); + return true; + } + + // Emits the structured body as its own function; call after the entry point is finished. + private bool TryEmitStructuredBodyFunction(out string error) + { + error = string.Empty; + if (_structuredPlan is not { } plan) + { + return true; + } + + _module.BeginFunction(_voidType, _module.TypeFunction(_voidType), _structuredBody); + _module.AddLabel(); + if (_functionScopeState) DeclareRegisterFiles(); + EmitInitialState(); + // Check bounds after initialization, in the same function as the registers. + var active = Load(_boolType, _programActive); + var execute = _module.AllocateId(); + var inactive = _module.AllocateId(); + _module.AddStatement(SpirvOp.SelectionMerge, execute, 0); + _module.AddStatement(SpirvOp.BranchConditional, active, execute, inactive); + _module.AddLabel(inactive); + _module.AddStatement(SpirvOp.Return); + _module.AddLabel(execute); + _structuredDry = false; + _structuredEmitted = new bool[plan.Blocks.Count]; + _copiedInstructions = 0; + _mergeCache = new(); + _blockLabels.Clear(); + if (!TryWalkSequence(plan, 0, null, null, atLoopHeader: false, out error)) + { + return false; + } + + _module.EndFunction(); + return true; + } + + private bool TryPlanStructuredControlFlow( + IReadOnlyList blocks, + out ControlFlowPlan plan, + out string reason) + { + plan = null!; + var count = blocks.Count; + var terminators = new BlockTerminator[count]; + for (var index = 0; index < count; index++) + { + if (!TryClassifyTerminator(blocks, index, out terminators[index])) + { + reason = $"unclassified terminator in block {index}"; + return false; + } + } + + var reachable = new bool[count]; + var order = ReversePostOrder(terminators, reachable); + var dominators = ComputeDominators(terminators, order, reachable); + + // Reducibility: every retreating edge must be a back edge to a dominating header. + var loops = new Dictionary(); + var position = new int[count]; + for (var index = 0; index < order.Count; index++) + { + position[order[index]] = index; + } + + foreach (var source in order) + { + foreach (var target in Successors(terminators[source])) + { + if (target == ExitNode || position[target] > position[source]) + { + continue; + } + + if (!Dominates(dominators, target, source)) + { + reason = $"irreducible edge {source}->{target}"; + return false; + } + + if (!loops.TryGetValue(target, out var loop)) + { + loop = new NaturalLoop { Header = target, Body = [target] }; + loops.Add(target, loop); + } + + CollectLoopBody(terminators, loop.Body, target, source); + } + } + + // Nest the loops and collect their exit blocks. + var ordered = loops.Values.OrderByDescending(static loop => loop.Body.Count).ToList(); + var innermost = new NaturalLoop?[count]; + foreach (var loop in ordered) + { + foreach (var block in loop.Body) + { + innermost[block] = loop; + } + } + + foreach (var loop in ordered) + { + loop.Parent = ordered + .Where(outer => outer != loop && outer.Body.IsSupersetOf(loop.Body)) + .OrderBy(static outer => outer.Body.Count) + .FirstOrDefault(); + var exits = new HashSet(); + foreach (var block in loop.Body) + { + foreach (var target in Successors(terminators[block])) + { + if (target != ExitNode && !loop.Body.Contains(target)) + { + exits.Add(target); + } + } + } + + loop.Exits = [.. exits.Order()]; + loop.Merge = exits.Count switch + { + 0 => ExitNode, + 1 => loop.Exits[0], + _ => MultiExitNode, + }; + } + + plan = new ControlFlowPlan + { + Blocks = blocks, + Terminators = terminators, + LoopsByHeader = loops, + InnermostLoop = innermost, + Reachable = reachable, + }; + + // Dry run: the same walk that emits, so emission cannot fail structurally. + _structuredDry = true; + _structuredEmitted = new bool[count]; + _copiedInstructions = 0; + _copyBudget = Math.Max(256, _request.Program.Instructions.Count); + _mergeCache = new(); + _blockLabels.Clear(); + if (!TryWalkSequence(plan, 0, null, null, atLoopHeader: false, out reason)) + { + return false; + } + + for (var index = 0; index < count; index++) + { + if (reachable[index] && !_structuredEmitted[index]) + { + reason = $"reachable block {index} was not placed"; + return false; + } + } + + reason = string.Empty; + return true; + } + + // Where a sequence ends: reaching `Block` means branching to `Label`, the merge block of + // the enclosing selection. + private readonly record struct StopTarget(int Block, uint Label); + + // Walks from `block` until control leaves the sequence: at `stop`, at a loop's continue or + // break, or at a return. Leaves the current SPIR-V block terminated. + private bool TryWalkSequence( + ControlFlowPlan plan, + int block, + StopTarget? stop, + LoopFrame? loop, + bool atLoopHeader, + out string error) + { + error = string.Empty; + while (true) + { + if (stop is { } stopTarget && block == stopTarget.Block) + { + Emit(SpirvOp.Branch, stopTarget.Label); + return true; + } + + if (block == ExitNode) + { + EmitReturn(); + return true; + } + + if (loop is not null && block == loop.Loop.Header && !atLoopHeader) + { + EmitBackEdge(loop); + return true; + } + + if (loop is not null && !loop.Loop.Body.Contains(block)) + { + // Leaving the loop: a loop with several exits records which one it took. + if (loop.Loop.Merge == MultiExitNode && !_structuredDry) + { + Store(loop.Loop.Selector, UInt((uint)loop.Loop.Exits.IndexOf(block))); + } + + Emit(SpirvOp.Branch, loop.BreakLabel); + return true; + } + + if (plan.LoopsByHeader.TryGetValue(block, out var nested) && !(atLoopHeader && loop?.Loop == nested)) + { + if (!TryWalkLoop(plan, nested, out error)) + { + return false; + } + + if (nested.Merge == ExitNode) + { + return true; + } + + if (nested.Merge != MultiExitNode) + { + block = nested.Merge; + continue; + } + + // Several exits: switch on the one the loop took. The switch merges where the + // exits meet, as a selection does. + if (!TryWalkArms(plan, nested.Header, nested.Exits, stop, loop, nested.Selector, out var next, out error)) + { + return false; + } + + if (next < 0) + { + return true; + } + + block = next; + continue; + } + + atLoopHeader = false; + uint label; + if (_structuredEmitted[block]) + { + // Reached from a second construct (arms that share a block their selection does + // not merge at): the path gets its own copy. Copies are bounded, since each can + // copy further shared blocks. + _copiedInstructions += plan.Blocks[block].EndIndex - plan.Blocks[block].StartIndex; + if (_copiedInstructions > _copyBudget) + { + error = $"copies of shared blocks exceed {_copyBudget} instructions"; + return false; + } + + label = AllocateLabel(); + } + else + { + _structuredEmitted[block] = true; + label = BlockLabel(block); + } + + Emit(SpirvOp.Branch, label); + EmitLabel(label); + if (!_structuredDry && !TryEmitBlockInstructions(plan.Blocks, block, out error)) + { + return false; + } + + var terminator = plan.Terminators[block]; + if (terminator.Kind == TerminatorKind.Exit) + { + EmitReturn(); + return true; + } + + if (terminator.Kind == TerminatorKind.Jump) + { + block = terminator.Taken; + continue; + } + + if (!TryWalkArms(plan, block, [terminator.Taken, terminator.Fallthrough], stop, loop, 0, out var merged, out error)) + { + return false; + } + + if (merged < 0) + { + return true; + } + + block = merged; + } + } + + // Emits a selection from `header` over `targets`: a conditional branch on the header's + // terminator, or, given a selector, a switch on the index of the target. Its merge block is + // synthetic: arms that reach the real merge block branch to it, and it then falls through + // to that block. `next` is the block the sequence continues at, or -1 when it has ended. + private bool TryWalkArms( + ControlFlowPlan plan, + int header, + IReadOnlyList targets, + StopTarget? stop, + LoopFrame? loop, + uint selector, + out int next, + out string error) + { + next = -1; + error = string.Empty; + var merge = FindMerge(plan, loop?.Loop, header); + if (merge.Virtual != VirtualExit.None && stop is { } outer) + { + // No block of its own to merge at, but inside a selection: arms that reach the + // enclosing merge go through this merge, which forwards to it. + merge = new MergeTarget(outer.Block, VirtualExit.None); + } + + var mergeLabel = AllocateLabel(); + var armLabels = targets.Select(_ => AllocateLabel()).ToArray(); + if (!_structuredDry) + { + // The condition is computed before the merge instruction, which must directly + // precede the branch. + if (selector != 0) + { + var operands = new List { Load(_uintType, selector), armLabels[0] }; + for (var index = 1; index < armLabels.Length; index++) + { + operands.Add((uint)index); + operands.Add(armLabels[index]); + } + + _module.AddStatement(SpirvOp.SelectionMerge, mergeLabel, 0); + _module.AddStatement(SpirvOp.Switch, [.. operands]); + } + else + { + if (!TryGetBranchCondition(plan.Terminators[header].Opcode, out var condition)) + { + error = $"no condition for {plan.Terminators[header].Opcode}"; + return false; + } + + _module.AddStatement(SpirvOp.SelectionMerge, mergeLabel, 0); + _module.AddStatement(SpirvOp.BranchConditional, condition, armLabels[0], armLabels[1]); + } + } + + // Arms stop at the real merge block. With a virtual merge every path leaves through a + // continue, break or return of its own, so the arms have no stop. + StopTarget? armStop = merge.Virtual == VirtualExit.None + ? new StopTarget(merge.Block, mergeLabel) + : null; + for (var index = 0; index < targets.Count; index++) + { + EmitLabel(armLabels[index]); + if (!TryWalkSequence(plan, targets[index], armStop, loop, atLoopHeader: false, out error)) + { + return false; + } + } + + EmitLabel(mergeLabel); + if (merge.Virtual != VirtualExit.None) + { + Emit(SpirvOp.Unreachable); + return true; + } + + // The merge is also the enclosing selection's merge: leave through its label. + if (stop is { } enclosing && merge.Block == enclosing.Block) + { + Emit(SpirvOp.Branch, enclosing.Label); + return true; + } + + next = merge.Block; + return true; + } + + private bool TryWalkLoop(ControlFlowPlan plan, NaturalLoop loop, out string error) + { + var header = AllocateLabel(); + var body = AllocateLabel(); + var frame = new LoopFrame + { + Loop = loop, + ContinueLabel = AllocateLabel(), + BreakLabel = AllocateLabel(), + }; + if (loop.Merge == MultiExitNode && !_structuredDry) + { + loop.Selector = _module.AddGlobalVariable( + _privateUintPointer, + SpirvStorageClass.Private, + _module.Constant(_uintType, 0)); + _interfaces.Add(loop.Selector); + } + + Emit(SpirvOp.Branch, header); + EmitLabel(header); + if (!_structuredDry) + { + _module.AddStatement(SpirvOp.LoopMerge, frame.BreakLabel, frame.ContinueLabel, 0); + _module.AddStatement(SpirvOp.Branch, body); + } + + EmitLabel(body); + if (!TryWalkSequence(plan, loop.Header, null, frame, atLoopHeader: true, out error)) + { + return false; + } + + // The continue construct: count the iteration for the step guard, then loop. + EmitLabel(frame.ContinueLabel); + Emit(SpirvOp.Branch, header); + EmitLabel(frame.BreakLabel); + if (!_structuredDry && _maxDispatcherSteps > 0) + { + // An exhausted step guard breaks out with the program inactive; it then ends the + // whole program, as the dispatcher's guard does. + var active = Load(_boolType, _programActive); + var stopLabel = AllocateLabel(); + var goOn = AllocateLabel(); + _module.AddStatement(SpirvOp.SelectionMerge, goOn, 0); + _module.AddStatement(SpirvOp.BranchConditional, active, goOn, stopLabel); + _module.AddLabel(stopLabel); + _module.AddStatement(SpirvOp.Return); + _module.AddLabel(goOn); + } + + if (loop.Merge == ExitNode) + { + Emit(SpirvOp.Unreachable); + } + + return true; + } + + private void EmitBackEdge(LoopFrame loop) + { + if (!_structuredDry && _maxDispatcherSteps > 0) + { + var steps = IAdd(Load(_uintType, _iterationGuard), UInt(1)); + Store(_iterationGuard, steps); + var withinLimit = _module.AddInstruction(SpirvOp.ULessThan, _boolType, steps, UInt((uint)_maxDispatcherSteps)); + var exhausted = AllocateLabel(); + var goOn = AllocateLabel(); + _module.AddStatement(SpirvOp.SelectionMerge, goOn, 0); + _module.AddStatement(SpirvOp.BranchConditional, withinLimit, goOn, exhausted); + _module.AddLabel(exhausted); + Store(_programActive, _module.ConstantBool(false)); + _module.AddStatement(SpirvOp.Branch, loop.BreakLabel); + _module.AddLabel(goOn); + } + + Emit(SpirvOp.Branch, loop.ContinueLabel); + } + + private void EmitReturn() + { + if (!_structuredDry) + { + Store(_programActive, _module.ConstantBool(false)); + } + + Emit(SpirvOp.Return); + } + + // The immediate post-dominator of `block` in its region. A region is a loop body (or the + // program), with inner loops collapsed to their headers, and the ways control leaves it + // (continue, break, return) as virtual nodes. + // + // Edges to those virtual nodes are left out when a node also has a real successor: a return, + // break or continue may leave a selection at any depth, so only the blocks the arms share + // decide the merge. A node whose paths only leave goes to its virtual nodes. + private MergeTarget FindMerge(ControlFlowPlan plan, NaturalLoop? region, int block) + { + if (_mergeCache!.TryGetValue((region, block), out var cached)) + { + return cached; + } + + // Nodes: region blocks, then virtual Return / Continue / Break, then the sink. + var members = new List(); + for (var index = 0; index < plan.Blocks.Count; index++) + { + if (plan.Reachable[index] && InRegion(plan, region, index)) + { + members.Add(index); + } + } + + var id = new Dictionary(); + for (var index = 0; index < members.Count; index++) + { + id[members[index]] = index; + } + + var returnNode = members.Count; + var continueNode = members.Count + 1; + var breakNode = members.Count + 2; + var sink = members.Count + 3; + var successors = new List[sink + 1]; + for (var index = 0; index <= sink; index++) + { + successors[index] = []; + } + + successors[returnNode].Add(sink); + successors[continueNode].Add(sink); + successors[breakNode].Add(sink); + foreach (var member in members) + { + var real = new List(); + var leaving = new List(); + foreach (var target in RegionSuccessors(plan, region, member)) + { + var node = target switch + { + ExitNode => returnNode, + _ when region is not null && target == region.Header => continueNode, + _ when region is not null && !region.Body.Contains(target) => breakNode, + _ => id.TryGetValue(target, out var inRegion) ? inRegion : breakNode, + }; + (node < members.Count ? real : leaving).Add(node); + } + + successors[id[member]].AddRange(real.Count != 0 ? real : leaving.Distinct()); + } + + var postDominators = ComputePostDominators(successors, sink); + var start = id[block]; + var merge = postDominators[start]; + if (merge >= members.Count) + { + // Every path leaves the region, but arms can still share blocks before leaving (an + // early return on one side). Those arms merge at the first block they share. + var shared = FindFirstSharedBlock(successors, members.Count, start); + if (shared >= 0) + { + merge = shared; + } + } + var result = merge == returnNode ? new MergeTarget(0, VirtualExit.Return) + : merge == continueNode ? new MergeTarget(0, VirtualExit.Continue) + : merge == breakNode ? new MergeTarget(0, VirtualExit.Break) + : merge == sink || merge < 0 ? new MergeTarget(0, VirtualExit.Mixed) + : new MergeTarget(members[merge], VirtualExit.None); + _mergeCache[(region, block)] = result; + return result; + } + + // The block reached from more than one successor of `start` that no other such block + // reaches, or -1. The region graph is acyclic (inner loops are collapsed), so reach sets + // are finite and "first" is well defined when one candidate precedes all others. + private static int FindFirstSharedBlock(List[] successors, int memberCount, int start) + { + var reachedBy = new int[memberCount]; + var arms = successors[start].Where(node => node < memberCount).Distinct().ToList(); + foreach (var arm in arms) + { + var seen = new bool[memberCount]; + var work = new Stack(); + seen[arm] = true; + work.Push(arm); + while (work.Count != 0) + { + var node = work.Pop(); + reachedBy[node]++; + foreach (var next in successors[node]) + { + if (next < memberCount && !seen[next]) + { + seen[next] = true; + work.Push(next); + } + } + } + } + + var candidates = Enumerable.Range(0, memberCount).Where(node => reachedBy[node] > 1).ToHashSet(); + if (candidates.Count == 0) + { + return -1; + } + + // Drop every candidate reached from another candidate; exactly one must remain. + var later = new HashSet(); + foreach (var candidate in candidates) + { + var work = new Stack(successors[candidate].Where(node => node < memberCount)); + while (work.Count != 0) + { + var node = work.Pop(); + if (later.Add(node)) + { + foreach (var next in successors[node]) + { + if (next < memberCount) + { + work.Push(next); + } + } + } + } + } + + var first = candidates.Where(candidate => !later.Contains(candidate)).ToList(); + return first.Count == 1 ? first[0] : -1; + } + + // A block belongs to a region when its innermost loop is the region itself, or when it is + // the header of a loop nested directly inside the region. + private static bool InRegion(ControlFlowPlan plan, NaturalLoop? region, int block) + { + var innermost = plan.InnermostLoop[block]; + if (innermost == region) + { + return true; + } + + return innermost is not null && innermost.Header == block && innermost.Parent == region; + } + + // Successors as the region sees them: a nested loop's header leads to that loop's merge. + private static IEnumerable RegionSuccessors(ControlFlowPlan plan, NaturalLoop? region, int block) + { + var innermost = plan.InnermostLoop[block]; + if (innermost is not null && innermost != region && innermost.Header == block) + { + if (innermost.Exits.Count == 0) + { + yield return ExitNode; + } + + foreach (var exit in innermost.Exits) + { + yield return exit; + } + + yield break; + } + + foreach (var target in Successors(plan.Terminators[block])) + { + yield return target; + } + } + + private static IEnumerable Successors(BlockTerminator terminator) + { + switch (terminator.Kind) + { + case TerminatorKind.Exit: + yield return ExitNode; + break; + case TerminatorKind.Jump: + yield return terminator.Taken; + break; + default: + yield return terminator.Taken; + if (terminator.Fallthrough != terminator.Taken) + { + yield return terminator.Fallthrough; + } + + break; + } + } + + private bool TryClassifyTerminator(IReadOnlyList blocks, int index, out BlockTerminator terminator) + { + var instructions = _request.Program.Instructions; + var last = instructions[blocks[index].EndIndex - 1]; + var fallthrough = index + 1 < blocks.Count ? index + 1 : ExitNode; + terminator = default; + if (last.Opcode == "SEndpgm") + { + terminator = new BlockTerminator(TerminatorKind.Exit, ExitNode, ExitNode, last.Opcode); + return true; + } + + if (last.Opcode == "SBranch" || last.Opcode.StartsWith("SCbranch", StringComparison.Ordinal)) + { + if (!TryGetBranchTargetPc(last, out var targetPc)) + { + return false; + } + + int target; + if (IsExitBranchTarget(instructions, targetPc)) + { + target = ExitNode; + } + else if (!TryFindBlock(blocks, targetPc, out target)) + { + return false; + } + + terminator = last.Opcode == "SBranch" || target == fallthrough + ? new BlockTerminator(TerminatorKind.Jump, target, target, last.Opcode) + : new BlockTerminator(TerminatorKind.Conditional, target, fallthrough, last.Opcode); + return true; + } + + terminator = new BlockTerminator(TerminatorKind.Jump, fallthrough, fallthrough, last.Opcode); + return true; + } + + private static List ReversePostOrder(BlockTerminator[] terminators, bool[] reachable) + { + var postOrder = new List(); + var stack = new Stack<(int Block, IEnumerator Next)>(); + reachable[0] = true; + stack.Push((0, Successors(terminators[0]).GetEnumerator())); + while (stack.Count != 0) + { + var (block, next) = stack.Peek(); + if (next.MoveNext()) + { + var target = next.Current; + if (target != ExitNode && !reachable[target]) + { + reachable[target] = true; + stack.Push((target, Successors(terminators[target]).GetEnumerator())); + } + + continue; + } + + stack.Pop(); + postOrder.Add(block); + } + + postOrder.Reverse(); + return postOrder; + } + + // Cooper, Harvey and Kennedy's iterative dominator algorithm over reverse post-order. + private static int[] ComputeDominators(BlockTerminator[] terminators, List order, bool[] reachable) + { + var count = terminators.Length; + var position = new int[count]; + for (var index = 0; index < order.Count; index++) + { + position[order[index]] = index; + } + + var predecessors = new List[count]; + for (var index = 0; index < count; index++) + { + predecessors[index] = []; + } + + foreach (var block in order) + { + foreach (var target in Successors(terminators[block])) + { + if (target != ExitNode) + { + predecessors[target].Add(block); + } + } + } + + var dominators = Enumerable.Repeat(-1, count).ToArray(); + dominators[order[0]] = order[0]; + var changed = true; + while (changed) + { + changed = false; + foreach (var block in order.Skip(1)) + { + var candidate = -1; + foreach (var predecessor in predecessors[block]) + { + if (dominators[predecessor] < 0) + { + continue; + } + + candidate = candidate < 0 ? predecessor : Intersect(dominators, position, candidate, predecessor); + } + + if (candidate >= 0 && dominators[block] != candidate) + { + dominators[block] = candidate; + changed = true; + } + } + } + + return dominators; + } + + private static int Intersect(int[] dominators, int[] position, int left, int right) + { + while (left != right) + { + while (position[left] > position[right]) + { + left = dominators[left]; + } + + while (position[right] > position[left]) + { + right = dominators[right]; + } + } + + return left; + } + + private static bool Dominates(int[] dominators, int dominator, int block) + { + while (true) + { + if (block == dominator) + { + return true; + } + + var next = dominators[block]; + if (next < 0 || next == block) + { + return false; + } + + block = next; + } + } + + private static void CollectLoopBody(BlockTerminator[] terminators, HashSet body, int header, int latch) + { + var predecessors = new Dictionary>(); + for (var index = 0; index < terminators.Length; index++) + { + foreach (var target in Successors(terminators[index])) + { + if (target == ExitNode) + { + continue; + } + + if (!predecessors.TryGetValue(target, out var list)) + { + predecessors[target] = list = []; + } + + list.Add(index); + } + } + + var work = new Stack(); + if (body.Add(latch)) + { + work.Push(latch); + } + + while (work.Count != 0) + { + var block = work.Pop(); + if (block == header || !predecessors.TryGetValue(block, out var list)) + { + continue; + } + + foreach (var predecessor in list) + { + if (body.Add(predecessor)) + { + work.Push(predecessor); + } + } + } + } + + // Immediate post-dominators on a small graph with a single sink, by the same iterative + // scheme run on the reversed edges. Returns -1 for nodes that cannot reach the sink. + private static int[] ComputePostDominators(List[] successors, int sink) + { + var count = successors.Length; + var predecessors = new List[count]; + for (var index = 0; index < count; index++) + { + predecessors[index] = []; + } + + for (var node = 0; node < count; node++) + { + foreach (var target in successors[node]) + { + predecessors[target].Add(node); + } + } + + // Post-order of the reversed graph from the sink. + var visited = new bool[count]; + var postOrder = new List(); + var stack = new Stack<(int Node, int Next)>(); + visited[sink] = true; + stack.Push((sink, 0)); + while (stack.Count != 0) + { + var (node, next) = stack.Pop(); + if (next < predecessors[node].Count) + { + stack.Push((node, next + 1)); + var predecessor = predecessors[node][next]; + if (!visited[predecessor]) + { + visited[predecessor] = true; + stack.Push((predecessor, 0)); + } + + continue; + } + + postOrder.Add(node); + } + + postOrder.Reverse(); + var position = new int[count]; + for (var index = 0; index < postOrder.Count; index++) + { + position[postOrder[index]] = index; + } + + var ipdom = Enumerable.Repeat(-1, count).ToArray(); + ipdom[sink] = sink; + var changed = true; + while (changed) + { + changed = false; + foreach (var node in postOrder.Skip(1)) + { + var candidate = -1; + foreach (var successor in successors[node]) + { + if (ipdom[successor] < 0) + { + continue; + } + + candidate = candidate < 0 ? successor : Intersect(ipdom, position, candidate, successor); + } + + if (candidate >= 0 && ipdom[node] != candidate) + { + ipdom[node] = candidate; + changed = true; + } + } + } + + return ipdom; + } + + private readonly Dictionary _blockLabels = new(); + + private uint BlockLabel(int block) + { + if (!_blockLabels.TryGetValue(block, out var label)) + { + label = _structuredDry ? 0u : _module.AllocateId(); + _blockLabels[block] = label; + } + + return label; + } + + private uint AllocateLabel() => _structuredDry ? 0u : _module.AllocateId(); + + private void EmitLabel(uint label) + { + if (!_structuredDry) + { + _module.AddLabel(label); + } + } + + private void Emit(SpirvOp op, params uint[] operands) + { + if (!_structuredDry) + { + _module.AddStatement(op, operands); + } + } + } +} diff --git a/src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvTranslator.Tessellation.cs b/src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvTranslator.Tessellation.cs new file mode 100644 index 0000000..4eee0cf --- /dev/null +++ b/src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvTranslator.Tessellation.cs @@ -0,0 +1,154 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +namespace SharpEmu.ShaderCompiler.Vulkan; + +public static partial class Gen5SpirvTranslator +{ + private sealed partial class CompilationContext + { + private uint _tessellationCoordinateInput; + private uint _tessellationPatchIdInput; + private uint _tessellationVertexIndexScratch; + + private uint TessellationWord(uint index) => LoadShaderDataDword(UInt(_request.Bindings.TessellationDataDword + index)); + private uint TessellationPointer(uint index) => Pair64(TessellationWord(index), TessellationWord(index + 1)); + + // Patches of one instance as a divisor; zero (a single instance) divides by all ones. + private uint PatchesPerInstanceDivisor() + { + var patches = TessellationWord(Gen5TessellationData.PatchesPerInstance); + return _module.AddInstruction(SpirvOp.Select, _uintType, + _module.AddInstruction(SpirvOp.IEqual, _boolType, patches, UInt(0)), UInt(uint.MaxValue), patches); + } + + private void EmitHullInputState() + { + var info = _request.TessellationHull!.Value; + var local = Load(_uintType, _localInvocationIndexInput); + var waveBase = BitwiseAnd(local, UInt(~63u)); + // Workgroup k runs hull group k of the dispatch: its patches, its off-chip slot + // (s2) and its run of the factor ring (s4). + var group = _module.AddInstruction(SpirvOp.CompositeExtract, _uintType, Load(_uvec3Type, _workGroupIdInput), 0); + var groupFirst = _module.AddInstruction(SpirvOp.IMul, _uintType, group, UInt(info.PatchesPerGroup)); + var firstPatch = IAdd(TessellationWord(Gen5TessellationData.FirstPatch), groupFirst); + var remainingPatches = _module.AddInstruction(SpirvOp.ISub, _uintType, + TessellationWord(Gen5TessellationData.PatchCount), groupFirst); + var patchCount = _module.AddInstruction(SpirvOp.Select, _uintType, + _module.AddInstruction(SpirvOp.ULessThan, _boolType, remainingPatches, UInt(info.PatchesPerGroup)), + remainingPatches, UInt(info.PatchesPerGroup)); + uint WaveCount(uint controlPoints) + { + var threads = _module.AddInstruction(SpirvOp.IMul, _uintType, patchCount, UInt(controlPoints)); + var remaining = _module.AddInstruction(SpirvOp.ISub, _uintType, threads, waveBase); + remaining = _module.AddInstruction(SpirvOp.Select, _uintType, + _module.AddInstruction(SpirvOp.ULessThan, _boolType, waveBase, threads), remaining, UInt(0)); + return _module.AddInstruction(SpirvOp.Select, _uintType, + _module.AddInstruction(SpirvOp.ULessThan, _boolType, remaining, UInt(64)), remaining, UInt(64)); + } + StoreS(2, IAdd(TessellationWord(Gen5TessellationData.OffchipOffset), _module.AddInstruction(SpirvOp.IMul, _uintType, + group, TessellationWord(Gen5TessellationData.OffchipSlotBytes)))); + StoreS(3, BitwiseOr(WaveCount(info.InputControlPoints), ShiftLeftLogical(WaveCount(info.OutputControlPoints), UInt(8)))); + StoreS(4, IAdd(TessellationWord(Gen5TessellationData.FactorOffset), _module.AddInstruction(SpirvOp.IMul, _uintType, + group, TessellationWord(Gen5TessellationData.FactorGroupBytes)))); + var relativePatch = _module.AddInstruction(SpirvOp.UDiv, _uintType, local, UInt(info.OutputControlPoints)); + var controlPoint = _module.AddInstruction(SpirvOp.UMod, _uintType, local, UInt(info.OutputControlPoints)); + // Patch numbers run across the instances of the draw; the guest sees the patch + // within its instance and the instance of each input control point. + var perInstance = PatchesPerInstanceDivisor(); + StoreV(0, _module.AddInstruction(SpirvOp.UMod, _uintType, IAdd(relativePatch, firstPatch), perInstance), false); + StoreV(1, BitwiseOr(relativePatch, ShiftLeftLogical(controlPoint, UInt(8))), false); + var inputPatch = IAdd(firstPatch, _module.AddInstruction(SpirvOp.UDiv, _uintType, local, UInt(info.InputControlPoints))); + var inputIndex = IAdd( + _module.AddInstruction(SpirvOp.IMul, _uintType, + _module.AddInstruction(SpirvOp.UMod, _uintType, inputPatch, perInstance), UInt(info.InputControlPoints)), + _module.AddInstruction(SpirvOp.UMod, _uintType, local, UInt(info.InputControlPoints))); + var indexSize = TessellationWord(Gen5TessellationData.IndexSize); + var vertex = _tessellationVertexIndexScratch; + Store(vertex, inputIndex); + var inputThreads = _module.AddInstruction(SpirvOp.IMul, _uintType, patchCount, UInt(info.InputControlPoints)); + var fetch = LogicalAnd(_module.AddInstruction(SpirvOp.INotEqual, _boolType, indexSize, UInt(0)), + _module.AddInstruction(SpirvOp.ULessThan, _boolType, local, inputThreads)); + EmitConditional(fetch, () => + { + var byteOffset = IAdd(_module.AddInstruction(SpirvOp.IMul, _uintType, inputIndex, indexSize), + TessellationWord(Gen5TessellationData.IndexByteOffset)); + var address = IAdd64(TessellationPointer(Gen5TessellationData.IndexAddress), Widen(BitwiseAnd(byteOffset, UInt(~3u)))); + var word = _module.AddInstruction(SpirvOp.Load, _uintType, DeviceWordPointer(address), 2, 4); + var shift = ShiftLeftLogical(BitwiseAnd(byteOffset, UInt(3)), UInt(3)); + var bits = ShiftLeftLogical(indexSize, UInt(3)); + var value = _module.AddInstruction(SpirvOp.BitFieldUExtract, _uintType, word, shift, bits); + Store(vertex, value); + }); + StoreV(2, IAdd(Load(_uintType, vertex), TessellationWord(Gen5TessellationData.VertexOffset)), false); + StoreV(3, local, false); + StoreV(5, IAdd(TessellationWord(Gen5TessellationData.InstanceId), + _module.AddInstruction(SpirvOp.UDiv, _uintType, inputPatch, perInstance)), false); + } + + private void DeclareTessellationInputs() + { + _module.AddCapability(SpirvCapability.Tessellation); + _tessellationCoordinateInput = _module.AddGlobalVariable( + _module.TypePointer(SpirvStorageClass.Input, _vec3Type), SpirvStorageClass.Input); + _module.AddDecoration(_tessellationCoordinateInput, SpirvDecoration.BuiltIn, (uint)SpirvBuiltIn.TessCoord); + _interfaces.Add(_tessellationCoordinateInput); + _tessellationPatchIdInput = _module.AddGlobalVariable( + _module.TypePointer(SpirvStorageClass.Input, _uintType), SpirvStorageClass.Input); + _module.AddDecoration(_tessellationPatchIdInput, SpirvDecoration.BuiltIn, (uint)SpirvBuiltIn.PrimitiveId); + _interfaces.Add(_tessellationPatchIdInput); + } + + private void EmitTessellationInputState() + { + // In the merged ES/GS ABI, five GS inputs precede the domain + // shader's U, V and relative patch ID. Coordinate values remain + // float bit patterns in the guest VGPR file. + var coordinates = Load(_vec3Type, _tessellationCoordinateInput); + StoreV(5, Bitcast(_uintType, _module.AddInstruction(SpirvOp.CompositeExtract, + _floatType, coordinates, 0)), guardWithExec: false); + StoreV(6, Bitcast(_uintType, _module.AddInstruction(SpirvOp.CompositeExtract, + _floatType, coordinates, 1)), guardWithExec: false); + // v7 is the patch within its hull group and v8 the patch ID. A native draw covers + // whole hull groups in order, so the native primitive ID gives the group (and its + // off-chip slot in s4) and the patch within it. + var drawPatch = Load(_uintType, _tessellationPatchIdInput); + if (!_request.Bindings.UsesTessellationData) + { + StoreV(7, drawPatch, guardWithExec: false); + StoreV(8, drawPatch, guardWithExec: false); + return; + } + + var groupPatches = TessellationWord(Gen5TessellationData.GroupPatches); + var divisor = _module.AddInstruction(SpirvOp.Select, _uintType, + _module.AddInstruction(SpirvOp.IEqual, _boolType, groupPatches, UInt(0)), UInt(uint.MaxValue), groupPatches); + var group = _module.AddInstruction(SpirvOp.UDiv, _uintType, drawPatch, divisor); + StoreV(7, _module.AddInstruction(SpirvOp.UMod, _uintType, drawPatch, divisor), guardWithExec: false); + StoreV(8, _module.AddInstruction(SpirvOp.UMod, _uintType, + IAdd(drawPatch, TessellationWord(Gen5TessellationData.FirstPatch)), PatchesPerInstanceDivisor()), guardWithExec: false); + StoreS(4, IAdd(TessellationWord(Gen5TessellationData.OffchipOffset), _module.AddInstruction(SpirvOp.IMul, _uintType, + group, TessellationWord(Gen5TessellationData.OffchipSlotBytes)))); + } + + private void EmitTessellationExecutionModes(uint entryPoint) + { + var info = _request.Tessellation!.Value; + _module.AddExecutionMode(entryPoint, info.Domain switch + { + Gen5TessellationDomain.Triangles => SpirvExecutionMode.Triangles, + Gen5TessellationDomain.Quads => SpirvExecutionMode.Quads, + _ => SpirvExecutionMode.Isolines, + }); + _module.AddExecutionMode(entryPoint, info.Spacing switch + { + Gen5TessellationSpacing.FractionalOdd => SpirvExecutionMode.SpacingFractionalOdd, + Gen5TessellationSpacing.FractionalEven => SpirvExecutionMode.SpacingFractionalEven, + _ => SpirvExecutionMode.SpacingEqual, + }); + _module.AddExecutionMode(entryPoint, info.Clockwise + ? SpirvExecutionMode.VertexOrderCw : SpirvExecutionMode.VertexOrderCcw); + if (info.PointMode) _module.AddExecutionMode(entryPoint, SpirvExecutionMode.PointMode); + } + } +} diff --git a/src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvTranslator.cs b/src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvTranslator.cs index 73a8a16..7cc4329 100644 --- a/src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvTranslator.cs +++ b/src/SharpEmu.ShaderCompiler.Vulkan/Gen5SpirvTranslator.cs @@ -67,9 +67,12 @@ public static partial class Gen5SpirvTranslator private readonly SpirvModuleBuilder _module = new(); private readonly Gen5SpirvStage _stage; + private bool HasVertexOutputs => _stage is Gen5SpirvStage.Vertex or Gen5SpirvStage.TessellationEvaluation; private readonly IReadOnlyList _pixelOutputBindings; private readonly bool _usesPixelValidMask; private readonly bool _enableGraphicsSubgroupOperations; + private readonly bool _nativeHalfConversionExact; + private readonly bool _zeroOutOfBoundsBufferReads; private readonly uint _waveLaneCount; private readonly bool _emulateWave64; @@ -127,6 +130,9 @@ public static partial class Gen5SpirvTranslator private readonly List _interfaces = []; private readonly Dictionary _pixelInputs = []; private readonly Dictionary _pixelOutputs = []; + private uint _pixelSampleMaskOutput; + private uint _pixelInvocationCoverageInput; + private uint _pixelDepthOutput; private readonly Dictionary _vertexOutputs = []; private readonly Dictionary _vertexInputsByPc = []; private uint _voidType; @@ -136,6 +142,10 @@ public static partial class Gen5SpirvTranslator private uint _longType; private uint _ulongType; private uint _floatType; + + // f16 and f16vec2, present only with SupportsExactFloat16Conversions. + private uint _halfType; + private uint _half2Type; private uint _vec2Type; private uint _vec3Type; private uint _vec4Type; @@ -146,15 +156,13 @@ public static partial class Gen5SpirvTranslator private uint _privateVec2Pointer; private readonly HashSet _flagVariables = []; private uint _runtimeBufferBiases; - // One Private variable per register the program touches, declared on first use. A whole - // register-file array is kept only for the VGPRs of a program that indexes them at run time - // (V_MOVREL*): MoltenVK lowers a 512-entry private array to thread memory, and every - // register access of every invocation then spills through it. + // Declare one variable per used register in the program's state storage class. + // This avoids spilling entire register-file arrays to per-thread memory. private readonly Dictionary _scalarRegisterVariables = new(); private readonly Dictionary _vectorRegisterVariables = new(); private uint _vectorRegisters; - private (uint First, uint Last) _dynamicVectorRange; - private uint _packedHalfRegisters; + private bool _needsDynamicVectorArray; + private readonly Dictionary _packedHalfRegisterVariables = []; private uint _scc; private uint _vcc; private uint _exec; @@ -172,9 +180,18 @@ public static partial class Gen5SpirvTranslator private uint _storageBlockPointer; private uint _storageUintPointer; private uint _lds; + private uint _ldsWritten; private uint _ldsElementPointer; private uint _lds64ElementPointer; private uint _ldsDwordMask; + // In-bounds condition of each workgroup LDS pointer that may point at the zero dword, + // so a store through it can be dropped when the guest address is out of range. + private readonly Dictionary _ldsPointerInBounds = []; + private bool _ldsIsWorkgroup; + // Compute LDS layout (DeclareLds): guest dwords below _ldsGuestDwordLimit, then the + // translator's dword kept at zero, then the wave64 exchange when the shader uses one. + private uint _ldsGuestDwordLimit; + private uint _ldsZeroDword; private uint _scratch; private uint _scratchElementPointer; private uint _scratchDwordCount; @@ -231,7 +248,9 @@ public static partial class Gen5SpirvTranslator SpirvImageDim Dimension = SpirvImageDim.Dim2D, uint ConversionFormat = 0, uint ShaderSwizzle = 0, - int EmulatedCompareFunction = -1); + int EmulatedCompareFunction = -1, + // The dense image resource index, which names this image's bit in the scaled-image mask. + uint ResourceIndex = 0); private readonly record struct SpirvVertexInput( uint Variable, @@ -246,7 +265,8 @@ public static partial class Gen5SpirvTranslator uint Variable, uint Type, Gen5PixelOutputKind Kind, - Gen5ColorComponentMapping ComponentMapping); + Gen5ColorComponentMapping ComponentMapping, + Gen5PixelExportFormat ExportFormat); public bool TryCompile(out Gen5SpirvShader shader, out string error) { @@ -254,6 +274,23 @@ public static partial class Gen5SpirvTranslator error = string.Empty; try { + if (_stage == Gen5SpirvStage.Pixel && _request.PixelDepthExportEnable && _request.EarlyFragmentTests) + { + error = "pixel depth exports require late fragment tests"; + return false; + } + if (_stage == Gen5SpirvStage.Pixel && _request.PixelSampleMaskExportEnable) + { + var exportSamples = _request.PixelMaskExportSamples; + var rasterSamples = _request.PixelRasterizationSamples; + if (exportSamples == 0 || rasterSamples == 0 || exportSamples > 16 || rasterSamples > 16 || + (exportSamples & (exportSamples - 1)) != 0 || (rasterSamples & (rasterSamples - 1)) != 0 || + (exportSamples != rasterSamples && !(exportSamples == 1 && rasterSamples == 2))) + { + error = $"unsupported pixel sample-mask export counts: export={exportSamples} raster={rasterSamples}"; + return false; + } + } if (Environment.GetEnvironmentVariable( "SHARPEMU_TRACE_TITLE_INTERFACE") == "1" && _request.Program.Address is 0x0000000500780000ul or @@ -287,9 +324,10 @@ public static partial class Gen5SpirvTranslator } var blocks = BuildBasicBlocks(_request.Program.Instructions); - // The fallback when the full structurer declines: blocks in program order behind a - // next-block guard, with natural loops as structured loops. - var structuredForward = StructuredForwardBlocks && blocks.Count != 0 && TryBuildLoopRegions(blocks, out _loopLatchByHeader); + // Initialize registers in the function that executes the program. + // The title-state diagnostic reads registers from the wrapper as well. + _functionScopeState = !(Environment.GetEnvironmentVariable("SHARPEMU_TRACE_TITLE_SHADER_STATE") == "1" && + _request.Program.Address == 0x0000000500781200ul); DeclareModule(); if (blocks.Count == 0) { @@ -297,11 +335,33 @@ public static partial class Gen5SpirvTranslator return false; } + PrepareControlFlow(blocks); + var functionType = _module.TypeFunction(_voidType); var main = _module.BeginFunction(_voidType, functionType); _module.AddName(main, "main"); _module.AddLabel(); + if (_functionScopeState && _structuredPlan is null) + { + DeclareRegisterFiles(); + } + if (_pixelInvocationCoverageInput != 0) + { + var coveragePointer = _module.AddInstruction(SpirvOp.AccessChain, + _module.TypePointer(SpirvStorageClass.Input, _intType), + _pixelInvocationCoverageInput, UInt(0)); + var coverage = Bitcast(_uintType, Load(_intType, coveragePointer)); + var eligible = BitwiseAnd(coverage, UInt(~_request.PixelShaderSampleExclusionMask)); + var executeLabel = _module.AllocateId(); + var excludedLabel = _module.AllocateId(); + var execute = _module.AddInstruction(SpirvOp.INotEqual, _boolType, eligible, UInt(0)); + _module.AddStatement(SpirvOp.SelectionMerge, executeLabel, 0); + _module.AddStatement(SpirvOp.BranchConditional, execute, executeLabel, excludedLabel); + _module.AddLabel(excludedLabel); + _module.AddStatement(SpirvOp.Kill); + _module.AddLabel(executeLabel); + } if (_stage == Gen5SpirvStage.Pixel && Environment.GetEnvironmentVariable( "SHARPEMU_FORCE_TITLE_EARLY_COLOR") == "1" && @@ -329,121 +389,14 @@ public static partial class Gen5SpirvTranslator _module.AddStatement(SpirvOp.Return); _module.AddLabel(); } - EmitInitialState(); + if (_structuredPlan is null) EmitInitialState(); - if (StructuredControlFlow && blocks.Count > 0 && TryStructureRange(blocks, 0, blocks.Count, blocks.Count, emit: false, out _)) + if (!TryEmitControlFlow(blocks, out error)) { - // As the dispatcher does before its first block: an out-of-bounds invocation runs nothing. - var runLabel = _module.AllocateId(); - var doneLabel = _module.AllocateId(); - var isActive = Load(_boolType, _programActive); - _module.AddStatement(SpirvOp.SelectionMerge, doneLabel, 0); - _module.AddStatement(SpirvOp.BranchConditional, isActive, runLabel, doneLabel); - _module.AddLabel(runLabel); - if (!TryStructureRange(blocks, 0, blocks.Count, blocks.Count, emit: true, out error)) - { - return false; - } - - _module.AddStatement(SpirvOp.Branch, doneLabel); - _module.AddLabel(doneLabel); - } - else - { - var loopHeader = _module.AllocateId(); - var switchHeader = _module.AllocateId(); - var switchMerge = _module.AllocateId(); - var loopContinue = _module.AllocateId(); - var loopMerge = _module.AllocateId(); - var defaultLabel = _module.AllocateId(); - var caseLabels = new uint[blocks.Count]; - for (var index = 0; index < caseLabels.Length; index++) - { - caseLabels[index] = _module.AllocateId(); + return false; } - if (structuredForward) - { - // Each invocation visits its blocks in program order, except for the back - // edges of natural loops. Emitting the blocks in order, each guarded by "this - // is the next block", and every loop as a structured SPIR-V loop runs exactly - // what the dispatcher loop would. The driver then sees ifs and loops and can - // keep only the live guest registers in hardware registers, where the - // dispatcher keeps nearly all of them live and pays a switch per block. - if (!TryEmitStructuredRange(blocks, 0, blocks.Count - 1, -1, out error)) - { - return false; - } - - _module.AddStatement(SpirvOp.Branch, loopMerge); - } - else - { - _module.AddStatement(SpirvOp.Branch, loopHeader); - _module.AddLabel(loopHeader); - // Check before the first block can write from an inactive invocation. - var programIsActive = Load(_boolType, _programActive); - _module.AddStatement(SpirvOp.LoopMerge, loopMerge, loopContinue, 0); - _module.AddStatement(SpirvOp.BranchConditional, programIsActive, switchHeader, loopMerge); - - _module.AddLabel(switchHeader); - var selector = Load(_uintType, _programCounter); - _module.AddStatement(SpirvOp.SelectionMerge, switchMerge, 0); - var switchOperands = new uint[2 + (blocks.Count * 2)]; - switchOperands[0] = selector; - switchOperands[1] = defaultLabel; - for (var index = 0; index < blocks.Count; index++) - { - switchOperands[2 + (index * 2)] = (uint)index; - switchOperands[3 + (index * 2)] = caseLabels[index]; - } - - _module.AddStatement(SpirvOp.Switch, switchOperands); - for (var index = 0; index < blocks.Count; index++) - { - _module.AddLabel(caseLabels[index]); - if (!TryEmitBlock(blocks, index, out error)) - { - error = $"block=0x{blocks[index].StartPc:X}: {error}"; - return false; - } - - _module.AddStatement(SpirvOp.Branch, switchMerge); - } - - _module.AddLabel(defaultLabel); - Store(_programActive, _module.ConstantBool(false)); - _module.AddStatement(SpirvOp.Branch, switchMerge); - - _module.AddLabel(switchMerge); - _module.AddStatement(SpirvOp.Branch, loopContinue); - _module.AddLabel(loopContinue); - var active = Load(_boolType, _programActive); - if (_maxDispatcherSteps > 0) - { - var steps = IAdd(Load(_uintType, _iterationGuard), UInt(1)); - Store(_iterationGuard, steps); - var withinLimit = _module.AddInstruction( - SpirvOp.ULessThan, - _boolType, - steps, - UInt((uint)_maxDispatcherSteps)); - active = _module.AddInstruction( - SpirvOp.LogicalAnd, - _boolType, - active, - withinLimit); - } - - _module.AddStatement( - SpirvOp.BranchConditional, - active, - loopHeader, - loopMerge); - } - - _module.AddLabel(loopMerge); - } + FlushPendingDeviceFaults(); if (_stage == Gen5SpirvStage.Pixel && Environment.GetEnvironmentVariable( "SHARPEMU_TRACE_TITLE_SHADER_STATE") == "1" && @@ -510,14 +463,36 @@ public static partial class Gen5SpirvTranslator _module.AddStatement(SpirvOp.Return); _module.EndFunction(); + if (!TryEmitStructuredBodyFunction(out error)) + { + return false; + } + + + + + EmitDeviceFormatLoadFunctions(); + EmitRuntimeLookupFunctions(); + var model = _stage switch { Gen5SpirvStage.Vertex => SpirvExecutionModel.Vertex, + Gen5SpirvStage.TessellationEvaluation => SpirvExecutionModel.TessellationEvaluation, Gen5SpirvStage.Pixel => SpirvExecutionModel.Fragment, _ => SpirvExecutionModel.GLCompute, }; _module.AddEntryPoint(model, main, "main", _interfaces); + if (_stage == Gen5SpirvStage.TessellationEvaluation) + EmitTessellationExecutionModes(main); + if (_request.SupportsExactFloat16Conversions) + { + // The f16 conversions round to nearest even and keep f16 denormals, as + // the hardware does. + _module.AddExecutionMode(main, SpirvExecutionMode.RoundingModeRTE, 16); + _module.AddExecutionMode(main, SpirvExecutionMode.DenormPreserve, 16); + _module.AddExecutionMode(main, SpirvExecutionMode.SignedZeroInfNanPreserve, 16); + } if (_request.ShaderSignedZeroInfNanPreserveFloat32Supported) { _module.AddExecutionMode( @@ -528,6 +503,12 @@ public static partial class Gen5SpirvTranslator if (_stage == Gen5SpirvStage.Pixel) { _module.AddExecutionMode(main, SpirvExecutionMode.OriginUpperLeft); + if (_request.EarlyFragmentTests) + _module.AddExecutionMode(main, SpirvExecutionMode.EarlyFragmentTests); + if (_pixelInvocationCoverageInput != 0) + _module.AddExecutionMode(main, SpirvExecutionMode.PostDepthCoverage); + if (_pixelDepthOutput != 0) + _module.AddExecutionMode(main, SpirvExecutionMode.DepthReplacing); } else if (_stage == Gen5SpirvStage.Compute) { @@ -546,7 +527,7 @@ public static partial class Gen5SpirvTranslator physicalSizes[2]); } - var attributeCount = _stage == Gen5SpirvStage.Vertex + var attributeCount = HasVertexOutputs ? (uint)_vertexOutputs.Count : (uint)_pixelInputs.Count; shader = new Gen5SpirvShader(_module.Build(), attributeCount); @@ -568,6 +549,13 @@ public static partial class Gen5SpirvTranslator _module.AddCapability(SpirvCapability.SignedZeroInfNanPreserve); } _module.AddCapability(SpirvCapability.Int64); + if (_request.SupportsExactFloat16Conversions) + { + _module.AddCapability(SpirvCapability.Float16); + _module.AddCapability(SpirvCapability.DenormPreserve); + _module.AddCapability(SpirvCapability.SignedZeroInfNanPreserve); + _module.AddCapability(SpirvCapability.RoundingModeRTE); + } _module.AddCapability(SpirvCapability.ImageQuery); if (UsesSubgroupOperations()) { @@ -595,6 +583,11 @@ public static partial class Gen5SpirvTranslator _ulongType = _module.TypeInt(64, signed: false); _floatType = _module.TypeFloat(32); _vec2Type = _module.TypeVector(_floatType, 2); + if (_request.SupportsExactFloat16Conversions) + { + _halfType = _module.TypeFloat(16); + _half2Type = _module.TypeVector(_halfType, 2); + } _vec3Type = _module.TypeVector(_floatType, 3); _vec4Type = _module.TypeVector(_floatType, 4); _uvec2Type = _module.TypeVector(_uintType, 2); @@ -605,77 +598,39 @@ public static partial class Gen5SpirvTranslator _privateVec2Pointer = _module.TypePointer(SpirvStorageClass.Private, _vec2Type); - if (_request.Program.Instructions.Any(static instruction => instruction.Opcode.StartsWith("VMovrel", StringComparison.Ordinal))) - { - // A dynamically indexed private array cannot live in registers: Metal spills the - // whole 2 KiB file to thread memory and every VGPR access becomes a memory access - // (Octopath's instanced vertex shader: 112 ms per draw). Keep one variable per - // register and resolve V_MOVREL* with a select over the registers the program uses. - _dynamicVectorRange = DynamicVectorRange(_request.Program); - } - - // Keep scalar state in uint variables, matching upstream's flag load/store conversion. - uint Flag(bool initialValue) - { - var variable = _module.AddGlobalVariable(_privateUintPointer, SpirvStorageClass.Private, UInt(initialValue ? 1u : 0u)); - _flagVariables.Add(variable); - return variable; - } - - _scc = Flag(false); - _vcc = Flag(false); - _exec = Flag(true); - _reachedPixelExport = Flag(false); + _moveRelativeOffsets = Ir.Gen5MoveRelativeOffsets.Analyze(_request.Program); + _needsDynamicVectorArray = _request.Program.Instructions.Any(instruction => + instruction.Opcode.StartsWith("VMovrel", StringComparison.Ordinal) && + !_moveRelativeOffsets.ContainsKey(instruction.Pc)); + // Register files belong to the function executing the program (see + // DeclareRegisterFiles). As Private globals, a file the driver fully promotes to + // registers stays behind as an unused .bss global, and AMD's driver crashes + // linking two stages that both carry one. + var registerStorage = _functionScopeState ? SpirvStorageClass.Function : SpirvStorageClass.Private; + _functionUintPointer = _module.TypePointer(registerStorage, _uintType); + _functionVec2Pointer = _module.TypePointer(registerStorage, _vec2Type); + // The per-invocation state (SCC, VCC, the block dispatcher's counters) also + // lives in that function: an unused Private global in a fragment shader crashes AMD's + // driver when it links the stages (seen with the dispatcher guard of a shader + // that no longer needs the dispatcher). + _functionBoolPointer = _module.TypePointer(registerStorage, _boolType); + // Only fragment validity is shared with the entry-point wrapper. Like every other + // flag it is kept in a uint (see Flag in DeclareRegisters). + var validityPointer = _module.TypePointer(SpirvStorageClass.Private, _uintType); + _exec = _module.AddGlobalVariable(validityPointer, SpirvStorageClass.Private, UInt(1)); + _flagVariables.Add(_exec); + _interfaces.Add(_exec); if (_usesPixelValidMask) { - _pixelValidMaskActive = Flag(true); - } - _programCounter = _module.AddGlobalVariable( - _privateUintPointer, - SpirvStorageClass.Private, - _module.Constant(_uintType, 0)); - _programActive = Flag(true); - if (UsesWave64Exchange()) - { - _wave64ExchangeParity = _module.AddGlobalVariable(_privateUintPointer, SpirvStorageClass.Private, UInt(0)); - _interfaces.Add(_wave64ExchangeParity); - _module.AddName(_wave64ExchangeParity, "wave64ExchangeParity"); - } - if (_maxDispatcherSteps > 0) - { - _iterationGuard = _module.AddGlobalVariable( - _privateUintPointer, - SpirvStorageClass.Private, - _module.Constant(_uintType, 0)); - _interfaces.Add(_iterationGuard); - _module.AddName(_iterationGuard, "pcGuard"); - } - - foreach (var slot in FindLaneSpillSlots()) - { - var variable = _module.AddGlobalVariable(_privateUintPointer, SpirvStorageClass.Private, _module.Constant(_uintType, 0)); - _interfaces.Add(variable); - _module.AddName(variable, $"v{slot.Register}_lane{slot.Lane}"); - _laneSpillSlots.Add(slot, variable); - } - - if (_vectorRegisters != 0) - { - _interfaces.Add(_vectorRegisters); - _module.AddName(_vectorRegisters, "vgpr"); - } - - _interfaces.Add(_scc); - _interfaces.Add(_vcc); - _interfaces.Add(_exec); - _interfaces.Add(_reachedPixelExport); - if (_pixelValidMaskActive != 0) - { + _pixelValidMaskActive = _module.AddGlobalVariable(validityPointer, SpirvStorageClass.Private, UInt(1)); + _flagVariables.Add(_pixelValidMaskActive); _interfaces.Add(_pixelValidMaskActive); _module.AddName(_pixelValidMaskActive, "pixelValidMaskActive"); } - _interfaces.Add(_programCounter); - _interfaces.Add(_programActive); + if (!_functionScopeState) + { + DeclareRegisterFiles(); + } { DeclareLayoutBindings(); @@ -716,6 +671,18 @@ public static partial class Gen5SpirvTranslator return; } + if (_request.CooperativeWave64Workgroup) + { + _wave64ExchangeElementPointer = _module.TypePointer(SpirvStorageClass.Workgroup, _uintType); + var array = _module.TypeArray(_uintType, Wave64ExchangeDwordCount * CooperativeWaveCount); + _wave64Exchange = _module.AddGlobalVariable( + _module.TypePointer(SpirvStorageClass.Workgroup, array), SpirvStorageClass.Workgroup); + _interfaces.Add(_wave64Exchange); + _module.AddName(_wave64Exchange, "wave64Exchange"); + DeclareCooperativeWaveScratch(); + return; + } + // Metal exposes 32 KiB of threadgroup memory on the Apple GPUs we // target. Some PS5 compute shaders legitimately request all of it, // so allocating another workgroup variable for the wave64 bridge @@ -728,9 +695,11 @@ public static partial class Gen5SpirvTranslator { _wave64Exchange = _lds; _wave64ExchangeElementPointer = _ldsElementPointer; - _wave64ExchangeOffset = _ldsDwordMask + 1 < LdsDwordCount - ? _ldsDwordMask + 1 - : LdsDwordCount - Wave64ExchangeDwordCount; + _wave64ExchangeOffset = _ldsIsWorkgroup + ? _ldsZeroDword + 1 + : _ldsDwordMask + 1 < LdsDwordCount + ? _ldsDwordMask + 1 + : LdsDwordCount - Wave64ExchangeDwordCount; return; } @@ -776,22 +745,45 @@ public static partial class Gen5SpirvTranslator ? ComputeLdsGuestDwordCount() : PrivateLdsDwordCount; _ldsDwordMask = dwordCount - 1; + _ldsIsWorkgroup = storageClass == SpirvStorageClass.Workgroup; var arrayDwordCount = UsesWave64Exchange() && dwordCount < LdsDwordCount ? dwordCount + Wave64ExchangeDwordCount : dwordCount; + if (_ldsIsWorkgroup) + { + // The translator's dwords follow the guest's: one kept at zero, which guest + // accesses past the declared size resolve to (the hardware reads 0 there and + // drops writes), then the wave64 exchange. A shader declaring the full 32 KiB + // keeps the array at that size: the reserved dwords take its top, and guest + // addresses there count as out of range. + var reserved = 1 + (UsesWave64Exchange() ? Wave64ExchangeDwordCount : 0); + arrayDwordCount = Math.Min(dwordCount + reserved, LdsDwordCount); + _ldsZeroDword = arrayDwordCount - reserved; + _ldsGuestDwordLimit = Math.Min(dwordCount, _ldsZeroDword); + } var ldsArrayType = _module.TypeArray(_uintType, arrayDwordCount); var ldsPointer = _module.TypePointer(storageClass, ldsArrayType); _ldsElementPointer = _module.TypePointer(storageClass, _uintType); _lds64ElementPointer = _module.TypePointer(storageClass, _ulongType); - _lds = storageClass == SpirvStorageClass.Workgroup - ? _module.AddGlobalVariable(ldsPointer, storageClass) - : _module.AddGlobalVariable( - ldsPointer, - storageClass, - _module.ConstantNull(ldsArrayType)); + // A private array must read zero where this invocation has not written, but + // zero-initializing it costs every invocation the whole array: Metal keeps a + // dynamically indexed private array in stack memory, so an NGG vertex shader + // cleared 8 KiB per vertex. A written-dword bitmap clears 1/32 of that, and + // LdsPointer zeroes a slot the first time it is addressed. + _lds = _module.AddGlobalVariable(ldsPointer, storageClass); _module.AddName(_lds, "lds"); _interfaces.Add(_lds); + if (storageClass == SpirvStorageClass.Private) + { + var bitmapType = _module.TypeArray(_uintType, (arrayDwordCount + 31) / 32); + _ldsWritten = _module.AddGlobalVariable( + _module.TypePointer(SpirvStorageClass.Private, bitmapType), + SpirvStorageClass.Private, + _module.ConstantNull(bitmapType)); + _module.AddName(_ldsWritten, "ldsWritten"); + _interfaces.Add(_ldsWritten); + } } internal static SpirvImageFormat DecodeStorageImageFormat( @@ -880,29 +872,36 @@ public static partial class Gen5SpirvTranslator } } - if (_stage == Gen5SpirvStage.Vertex) + if (HasVertexOutputs) { - DeclareVertexInputs(); + if (_stage == Gen5SpirvStage.Vertex) + { + DeclareVertexInputs(); - var inputPointer = - _module.TypePointer(SpirvStorageClass.Input, _uintType); - _vertexIndexInput = _module.AddGlobalVariable( - inputPointer, - SpirvStorageClass.Input); - _module.AddDecoration( - _vertexIndexInput, - SpirvDecoration.BuiltIn, - (uint)SpirvBuiltIn.VertexIndex); - _interfaces.Add(_vertexIndexInput); + var inputPointer = + _module.TypePointer(SpirvStorageClass.Input, _uintType); + _vertexIndexInput = _module.AddGlobalVariable( + inputPointer, + SpirvStorageClass.Input); + _module.AddDecoration( + _vertexIndexInput, + SpirvDecoration.BuiltIn, + (uint)SpirvBuiltIn.VertexIndex); + _interfaces.Add(_vertexIndexInput); - _instanceIndexInput = _module.AddGlobalVariable( - inputPointer, - SpirvStorageClass.Input); - _module.AddDecoration( - _instanceIndexInput, - SpirvDecoration.BuiltIn, - (uint)SpirvBuiltIn.InstanceIndex); - _interfaces.Add(_instanceIndexInput); + _instanceIndexInput = _module.AddGlobalVariable( + inputPointer, + SpirvStorageClass.Input); + _module.AddDecoration( + _instanceIndexInput, + SpirvDecoration.BuiltIn, + (uint)SpirvBuiltIn.InstanceIndex); + _interfaces.Add(_instanceIndexInput); + } + else + { + DeclareTessellationInputs(); + } var outputPointer = _module.TypePointer(SpirvStorageClass.Output, _vec4Type); @@ -1042,6 +1041,10 @@ public static partial class Gen5SpirvTranslator { _module.AddDecoration(variable, SpirvDecoration.Flat); } + else + { + DecorateSampleInterpolant(variable); + } _pixelInputs.Add(attribute, variable); _interfaces.Add(variable); @@ -1061,6 +1064,35 @@ public static partial class Gen5SpirvTranslator (uint)SpirvBuiltIn.FragCoord); _interfaces.Add(_fragCoordInput); DeclarePixelSystemInputs(); + if (_request.EarlyFragmentTests && (_request.PixelShaderSampleExclusionMask & 0xFFFFu) != 0) + { + _module.AddExtension("SPV_KHR_post_depth_coverage"); + _module.AddCapability(SpirvCapability.SampleMaskPostDepthCoverage); + _module.AddCapability(SpirvCapability.SampleRateShading); + _pixelInvocationCoverageInput = _module.AddGlobalVariable( + _module.TypePointer(SpirvStorageClass.Input, _module.TypeArray(_intType, 1)), + SpirvStorageClass.Input); + _module.AddDecoration(_pixelInvocationCoverageInput, SpirvDecoration.BuiltIn, + (uint)SpirvBuiltIn.SampleMask); + _interfaces.Add(_pixelInvocationCoverageInput); + } + if (_request.PixelDepthExportEnable) + { + _pixelDepthOutput = _module.AddGlobalVariable( + _module.TypePointer(SpirvStorageClass.Output, _floatType), SpirvStorageClass.Output); + _module.AddDecoration(_pixelDepthOutput, SpirvDecoration.BuiltIn, + (uint)SpirvBuiltIn.FragDepth); + _interfaces.Add(_pixelDepthOutput); + } + if (_request.PixelSampleMaskExportEnable) + { + var maskArray = _module.TypeArray(_intType, 1); + _pixelSampleMaskOutput = _module.AddGlobalVariable( + _module.TypePointer(SpirvStorageClass.Output, maskArray), SpirvStorageClass.Output); + _module.AddDecoration(_pixelSampleMaskOutput, SpirvDecoration.BuiltIn, + (uint)SpirvBuiltIn.SampleMask); + _interfaces.Add(_pixelSampleMaskOutput); + } var declaredPixelOutputs = Environment.GetEnvironmentVariable( @@ -1081,13 +1113,18 @@ public static partial class Gen5SpirvTranslator variable, SpirvDecoration.Location, binding.HostLocation); + if (binding.Index != 0) + { + _module.AddDecoration(variable, SpirvDecoration.Index, binding.Index); + } _pixelOutputs.Add( binding.ExportTarget, new SpirvPixelOutput( variable, outputType, binding.Kind, - binding.ComponentMapping)); + binding.ComponentMapping, + binding.ExportFormat)); _interfaces.Add(variable); } } @@ -1207,7 +1244,7 @@ public static partial class Gen5SpirvTranslator // vertex count in bits 0-7, primitive count in bits 8-15. Left at zero, EXEC // becomes all 64 lanes, and a waterfall loop then waits forever for lanes that // do not exist to clear their bits (Astro Bot intro_next froze the GPU this way). - if (_stage == Gen5SpirvStage.Vertex && _request.UserDataBase > 3) + if (HasVertexOutputs && _request.UserDataBase > 3) { uint laneCount; if (_subgroupInvocationIdInput == 0) @@ -1227,11 +1264,26 @@ public static partial class Gen5SpirvTranslator Store(_programCounter, UInt(0)); Store(_programActive, _module.ConstantBool(true)); - - if (_stage == Gen5SpirvStage.Vertex) + if (_ldsIsWorkgroup) { - StoreV(5, Load(_uintType, _vertexIndexInput), guardWithExec: false); - StoreV(8, Load(_uintType, _instanceIndexInput), guardWithExec: false); + // Every invocation zeroes the dword out-of-range LDS reads resolve to. Only zero + // is ever stored there, so each invocation sees zero without a barrier. + Store( + _module.AddInstruction(SpirvOp.AccessChain, _ldsElementPointer, _lds, UInt(_ldsZeroDword)), + UInt(0)); + } + + if (HasVertexOutputs) + { + if (_stage == Gen5SpirvStage.Vertex) + { + StoreV(5, Load(_uintType, _vertexIndexInput), guardWithExec: false); + StoreV(8, Load(_uintType, _instanceIndexInput), guardWithExec: false); + } + else + { + EmitTessellationInputState(); + } // Give every declared param output a defined starting value. // Outputs the program actually exports overwrite this; the @@ -1259,12 +1311,16 @@ public static partial class Gen5SpirvTranslator } else if (_stage == Gen5SpirvStage.Pixel) { - var fragCoord = Load(_vec4Type, _fragCoordInput); + var fragCoord = ScalePixelPosition(Load(_vec4Type, _fragCoordInput)); EmitPixelInputState(fragCoord); + if (_pixelDepthOutput != 0) + Store(_pixelDepthOutput, _module.AddInstruction(SpirvOp.CompositeExtract, _floatType, fragCoord, 2)); foreach (var output in _pixelOutputs.Values) { Store(output.Variable, _module.ConstantNull(output.Type)); } + if (_pixelSampleMaskOutput != 0) + Store(PixelSampleMaskOutputElement(), Bitcast(_intType, UInt(uint.MaxValue))); } else { @@ -1340,6 +1396,7 @@ public static partial class Gen5SpirvTranslator UInt(checked(_localSizeX * _localSizeY * _localSizeZ))); } } + if (_request.TessellationHull is not null) EmitHullInputState(); } } @@ -1442,21 +1499,176 @@ public static partial class Gen5SpirvTranslator private enum SharedMemoryPhase { None, Read, Write } + // The PC dispatcher: one loop whose switch runs the block the program counter names. + // Correct for any control flow, including irreducible graphs, but costly to compile. + private bool TryEmitDispatcher(IReadOnlyList blocks, out string error) + { + error = string.Empty; + var loopHeader = _module.AllocateId(); + var switchHeader = _module.AllocateId(); + var switchMerge = _module.AllocateId(); + var loopContinue = _module.AllocateId(); + var loopMerge = _module.AllocateId(); + var defaultLabel = _module.AllocateId(); + var caseLabels = new uint[blocks.Count]; + for (var index = 0; index < caseLabels.Length; index++) + { + caseLabels[index] = _module.AllocateId(); + } + + _module.AddStatement(SpirvOp.Branch, loopHeader); + _module.AddLabel(loopHeader); + // Check before the first block can write from an inactive invocation. + var programIsActive = Load(_boolType, _programActive); + _module.AddStatement(SpirvOp.LoopMerge, loopMerge, loopContinue, 0); + _module.AddStatement(SpirvOp.BranchConditional, programIsActive, switchHeader, loopMerge); + + _module.AddLabel(switchHeader); + var selector = Load(_uintType, _programCounter); + if (blocks.Count <= 512) + { + _module.AddStatement(SpirvOp.SelectionMerge, switchMerge, 0); + var switchOperands = new uint[2 + (blocks.Count * 2)]; + switchOperands[0] = selector; + switchOperands[1] = defaultLabel; + for (var index = 0; index < blocks.Count; index++) + { + switchOperands[2 + (index * 2)] = (uint)index; + switchOperands[3 + (index * 2)] = caseLabels[index]; + } + + _module.AddStatement(SpirvOp.Switch, switchOperands); + for (var index = 0; index < blocks.Count; index++) + { + _module.AddLabel(caseLabels[index]); + if (!TryEmitBlock(blocks, index, out error)) + { + error = $"block=0x{blocks[index].StartPc:X}: {error}"; + return false; + } + + _module.AddStatement(SpirvOp.Branch, switchMerge); + } + + _module.AddLabel(defaultLabel); + Store(_programActive, _module.ConstantBool(false)); + _module.AddStatement(SpirvOp.Branch, switchMerge); + } + else + { + var dispatchRoot = _module.AllocateId(); + var dispatchMerge = _module.AllocateId(); + var inRange = _module.AddInstruction( + SpirvOp.ULessThan, + _boolType, + selector, + UInt((uint)blocks.Count)); + _module.AddStatement(SpirvOp.SelectionMerge, dispatchMerge, 0); + _module.AddStatement(SpirvOp.BranchConditional, inRange, dispatchRoot, defaultLabel); + _module.AddLabel(dispatchRoot); + + bool EmitDispatchTree(int first, int count, uint continuation, out string treeError) + { + treeError = string.Empty; + if (count == 1) + { + _module.AddStatement(SpirvOp.Branch, caseLabels[first]); + _module.AddLabel(caseLabels[first]); + if (!TryEmitBlock(blocks, first, out treeError)) + { + treeError = $"block=0x{blocks[first].StartPc:X}: {treeError}"; + return false; + } + + _module.AddStatement(SpirvOp.Branch, continuation); + return true; + } + + var leftCount = count / 2; + var split = first + leftCount; + var leftLabel = _module.AllocateId(); + var rightLabel = _module.AllocateId(); + var mergeLabel = _module.AllocateId(); + var left = _module.AddInstruction( + SpirvOp.ULessThan, + _boolType, + selector, + UInt((uint)split)); + _module.AddStatement(SpirvOp.SelectionMerge, mergeLabel, 0); + _module.AddStatement(SpirvOp.BranchConditional, left, leftLabel, rightLabel); + _module.AddLabel(leftLabel); + if (!EmitDispatchTree(first, leftCount, mergeLabel, out treeError)) + { + return false; + } + + _module.AddLabel(rightLabel); + if (!EmitDispatchTree(split, count - leftCount, mergeLabel, out treeError)) + { + return false; + } + + _module.AddLabel(mergeLabel); + _module.AddStatement(SpirvOp.Branch, continuation); + return true; + } + + if (!EmitDispatchTree(0, blocks.Count, dispatchMerge, out error)) + { + return false; + } + + _module.AddLabel(defaultLabel); + Store(_programActive, _module.ConstantBool(false)); + _module.AddStatement(SpirvOp.Branch, dispatchMerge); + _module.AddLabel(dispatchMerge); + _module.AddStatement(SpirvOp.Branch, switchMerge); + } + + _module.AddLabel(switchMerge); + _module.AddStatement(SpirvOp.Branch, loopContinue); + _module.AddLabel(loopContinue); + var active = Load(_boolType, _programActive); + if (_maxDispatcherSteps > 0) + { + var steps = IAdd(Load(_uintType, _iterationGuard), UInt(1)); + Store(_iterationGuard, steps); + var withinLimit = _module.AddInstruction( + SpirvOp.ULessThan, + _boolType, + steps, + UInt((uint)_maxDispatcherSteps)); + active = _module.AddInstruction( + SpirvOp.LogicalAnd, + _boolType, + active, + withinLimit); + } + + _module.AddStatement( + SpirvOp.BranchConditional, + active, + loopHeader, + loopMerge); + _module.AddLabel(loopMerge); + return true; + } + private bool TryEmitBlock( IReadOnlyList blocks, int blockIndex, out string error) { - if (!TryEmitBlockBody(blocks, blockIndex, out error)) + if (!TryEmitBlockInstructions(blocks, blockIndex, out error)) { return false; } - return TryEmitBlockTerminator(blocks, blockIndex, out error); + return TryEmitDispatcherTerminator(blocks, blockIndex, out error); } - // Every instruction of a block except its terminating branch or s_endpgm. - private bool TryEmitBlockBody( + // Emits every instruction of a block except its terminating branch or S_ENDPGM. + private bool TryEmitBlockInstructions( IReadOnlyList blocks, int blockIndex, out string error) @@ -1465,10 +1677,14 @@ public static partial class Gen5SpirvTranslator var block = blocks[blockIndex]; var halfMaskPlan = HalfMaskPlan(); // One guest wave can span two host subgroups. Keep its shared-memory phases ordered. - // The half-mask plan inserts its own barriers; other wave64 programs also - // need LDS phase ordering when reads and writes span basic blocks. - var synchronizeSharedMemory = _emulateWave64 && halfMaskPlan is null; + // The half-mask plan's barriers only order its own mask exchanges, not the program's + // LDS traffic: with phase ordering off under the plan, UE's FFT bloom (wave64, 512 + // threads, LDS butterflies) read half-written rows and blurred Silent Hill's whole + // post chain into a flat haze. + var synchronizeSharedMemory = _emulateWave64; var sharedMemoryPhase = SharedMemoryPhase.None; + // A wait in another block may or may not have run before this one. + _vectorMemoryWaited = null; for (var index = block.StartIndex; index < block.EndIndex; index++) { var instruction = _request.Program.Instructions[index]; @@ -1504,6 +1720,50 @@ public static partial class Gen5SpirvTranslator } } + // A run of plain vector ALU writes shares one EXEC test: inside the branch no lane + // is masked off, so each write stores straight into its register instead of + // selecting against the old value, and the old values stop being live across the + // whole run. Only the join needs them. + var runLength = ExecRunLength(block, halfMaskPlan, index); + if (runLength >= ExecRunMinimum) + { + var last = index + runLength - 1; + var failed = false; + var runFailure = string.Empty; + EmitConditional(Load(_boolType, _exec), () => + { + for (var inner = index; inner <= last && !failed; inner++) + { + var run = _request.Program.Instructions[inner]; + _execKnownFull = true; + _emittingPc = run.Pc; + var ok = TryEmitInstruction(run, out var runError); + _emittingPc = null; + _execKnownFull = false; + if (!ok) + { + runFailure = $"pc=0x{run.Pc:X} {run.Opcode}: {runError}"; + failed = true; + return; + } + + CapturePixelVgprs(run); + CapturePixelVgprPoints(run); + MarkPixelPath(run); + CapturePixelExec(run); + } + }); + + if (failed) + { + error = runFailure; + return false; + } + + index = last; + continue; + } + _execKnownFull = IsExecKnownFull(instruction.Pc); _emittingPc = instruction.Pc; var emitted = TryEmitInstruction(instruction, out error); @@ -1525,8 +1785,61 @@ public static partial class Gen5SpirvTranslator return true; } - // Stores the next dispatcher block (or ends the program) for a block's terminator. - private bool TryEmitBlockTerminator( + // SHARPEMU_EXEC_RUN=0 turns the shared EXEC test off; SHARPEMU_EXEC_RUN_MIN sets how many + // instructions a run needs before the branch pays for itself. + private static readonly bool ExecRunGrouping = + Environment.GetEnvironmentVariable("SHARPEMU_EXEC_RUN") != "0"; + + private static readonly int ExecRunMinimum = + ExecRunGrouping && int.TryParse(Environment.GetEnvironmentVariable("SHARPEMU_EXEC_RUN_MIN"), out var minimum) && minimum >= 2 + ? minimum + : ExecRunGrouping ? DefaultExecRunMinimum : int.MaxValue; + + // Six measured best on Silent Hill's volumetric fog: shorter runs pay for the branch more + // often than they save, longer ones leave the short runs guarded. + private const int DefaultExecRunMinimum = 6; + + // How many instructions from startIndex may share one EXEC test. The run stops at anything + // the wave as a whole has to run even where this lane is masked off: every scalar result + // (SALU is not EXEC masked on the hardware, and each invocation keeps its own copy of the + // scalar registers), every cross-lane read, every memory or LDS access, and every point the + // half-mask plan wants a mask exchange or a barrier emitted before. + private int ExecRunLength(ShaderBlock block, Ir.Gen5Wave64HalfMaskPlan? halfMaskPlan, int startIndex) + { + // Compute only. The same grouping measured 28% slower on Silent Hill's deferred + // lighting pixel program (11.0 -> 14.1 ms per scope): a pixel wave already runs with + // helper lanes and an extra branch per run costs it more than the selects it removes. + if (!ExecRunGrouping || _stage != Gen5SpirvStage.Compute) + { + return 0; + } + + var instructions = _request.Program.Instructions; + var length = 0; + for (var index = startIndex; index < block.EndIndex; index++) + { + var instruction = instructions[index]; + if (length != 0 && halfMaskPlan is not null && + (halfMaskPlan.ExactPairsBefore.ContainsKey(instruction.Pc) || + halfMaskPlan.SharedMemoryBarriersBefore.Contains(instruction.Pc))) + { + break; + } + + if (!Ir.Gen5ExecRunAnalysis.IsExecMaskedVectorAlu(instruction)) + { + break; + } + + length++; + } + + return length; + } + + + // Selects the next block through the program counter; the dispatcher loop branches on it. + private bool TryEmitDispatcherTerminator( IReadOnlyList blocks, int blockIndex, out string error) @@ -1610,482 +1923,6 @@ public static partial class Gen5SpirvTranslator return true; } - // Forward-only, properly nested control flow is emitted as structured selections instead of - // the PC-dispatch loop: a loop around a switch keeps every register live across iterations, - // blocks cannot be optimized together, and the Metal compiler spills. SHARPEMU_STRUCTURED_CF=0 - // keeps the dispatcher for every program. - private static readonly bool StructuredControlFlow = - Environment.GetEnvironmentVariable("SHARPEMU_STRUCTURED_CF") != "0"; - - private static bool IsStructuredCondition(string opcode) => opcode is - "SCbranchScc0" or "SCbranchScc1" or "SCbranchVccz" or "SCbranchVccnz" or - "SCbranchExecz" or "SCbranchExecnz" or - "SCbranchCdbgsys" or "SCbranchCdbguser" or "SCbranchCdbgsysOrUser" or "SCbranchCdbgsysAndUser"; - - // The block a branch continues at, or blocks.Count for a branch past the last instruction. - private bool TryResolveBranchBlock(IReadOnlyList blocks, Gen5ShaderInstruction branch, out int target) - { - target = -1; - if (!TryGetBranchTargetPc(branch, out var targetPc)) - { - return false; - } - - if (IsExitBranchTarget(_request.Program.Instructions, targetPc)) - { - target = blocks.Count; - return true; - } - - return TryFindBlock(blocks, targetPc, out target); - } - - // Emits (or, without emit, only checks) blocks [begin, end). A branch from the range's last - // block to leave falls through to the range end. Inside a loop body, latch is the block - // whose backward branch the caller emits as the loop's continue, and a branch to end would - // leave the loop, which a structured body cannot express. - private bool TryStructureRange( - IReadOnlyList blocks, - int begin, - int end, - int leave, - bool emit, - out string error, - int latch = -1) - { - error = string.Empty; - var instructions = _request.Program.Instructions; - var index = begin; - while (index < end) - { - // A loop body starts at its own header, which must not open the loop again. - if (!(latch >= 0 && index == begin) && index != latch && - TryFindLoopLatch(blocks, index, end, out var loopLatch)) - { - if (!TryStructureLoop(blocks, index, loopLatch, emit, out error)) - { - return false; - } - - index = loopLatch + 1; - continue; - } - - if (emit && !TryEmitBlockBody(blocks, index, out error)) - { - error = $"block=0x{blocks[index].StartPc:X}: {error}"; - return false; - } - - if (index == latch) - { - return true; - } - - var terminator = instructions[blocks[index].EndIndex - 1]; - var next = index + 1; - if (terminator.Opcode == "SEndpgm") - { - if (next != blocks.Count) - { - error = "s_endpgm before the last block"; - return false; - } - - if (emit) - { - Store(_programActive, _module.ConstantBool(false)); - } - - index = next; - continue; - } - - if (terminator.Opcode == "SBranch") - { - if (!TryResolveBranchBlock(blocks, terminator, out var branchTarget) || - !(branchTarget == next || (next == end && branchTarget == leave))) - { - error = $"unstructured s_branch at 0x{terminator.Pc:X}"; - return false; - } - - index = next; - continue; - } - - if (!terminator.Opcode.StartsWith("SCbranch", StringComparison.Ordinal)) - { - index = next; - continue; - } - - if (!IsStructuredCondition(terminator.Opcode) || - !TryResolveBranchBlock(blocks, terminator, out var target) || target <= index) - { - error = $"unstructured {terminator.Opcode} at 0x{terminator.Pc:X}"; - return false; - } - - // if (taken) break; -- only out of the innermost loop, to the block after its latch. - if (_structuredLoopExit >= 0 && target == _structuredLoopExit) - { - if (emit) - { - TryGetBranchCondition(terminator.Opcode, out var leaveLoop); - var breakLabel = _module.AllocateId(); - var stayLabel = _module.AllocateId(); - _module.AddStatement(SpirvOp.SelectionMerge, stayLabel, 0); - _module.AddStatement(SpirvOp.BranchConditional, leaveLoop, breakLabel, stayLabel); - _module.AddLabel(breakLabel); - _module.AddStatement(SpirvOp.Branch, _structuredLoopMerge); - _module.AddLabel(stayLabel); - } - - index = next; - continue; - } - - if (target > end || (latch >= 0 && target == end)) - { - error = $"unstructured {terminator.Opcode} at 0x{terminator.Pc:X}"; - return false; - } - - if (target == next) - { - index = next; - continue; - } - - // if/else: the skipped range ends with an s_branch over the taken range. - var thenLast = instructions[blocks[target - 1].EndIndex - 1]; - if (target < blocks.Count && target - 1 > index && thenLast.Opcode == "SBranch" && - TryResolveBranchBlock(blocks, thenLast, out var joinTarget) && joinTarget > target && joinTarget <= end && - !(latch >= 0 && joinTarget == end)) - { - if (!emit) - { - if (!TryStructureRange(blocks, next, target, joinTarget, emit: false, out error) || - !TryStructureRange(blocks, target, joinTarget, joinTarget, emit: false, out error)) - { - return false; - } - } - else - { - TryGetBranchCondition(terminator.Opcode, out var taken); - var thenLabel = _module.AllocateId(); - var elseLabel = _module.AllocateId(); - var joinLabel = _module.AllocateId(); - _module.AddStatement(SpirvOp.SelectionMerge, joinLabel, 0); - _module.AddStatement(SpirvOp.BranchConditional, taken, elseLabel, thenLabel); - _module.AddLabel(thenLabel); - if (!TryStructureRange(blocks, next, target, joinTarget, emit: true, out error)) - { - return false; - } - - _module.AddStatement(SpirvOp.Branch, joinLabel); - _module.AddLabel(elseLabel); - if (!TryStructureRange(blocks, target, joinTarget, joinTarget, emit: true, out error)) - { - return false; - } - - _module.AddStatement(SpirvOp.Branch, joinLabel); - _module.AddLabel(joinLabel); - } - - index = joinTarget; - continue; - } - - // if (!taken) { skipped range } - if (!emit) - { - if (!TryStructureRange(blocks, next, target, target, emit: false, out error)) - { - return false; - } - } - else - { - TryGetBranchCondition(terminator.Opcode, out var taken); - var bodyLabel = _module.AllocateId(); - var mergeLabel = _module.AllocateId(); - _module.AddStatement(SpirvOp.SelectionMerge, mergeLabel, 0); - _module.AddStatement(SpirvOp.BranchConditional, taken, mergeLabel, bodyLabel); - _module.AddLabel(bodyLabel); - if (!TryStructureRange(blocks, next, target, target, emit: true, out error)) - { - return false; - } - - _module.AddStatement(SpirvOp.Branch, mergeLabel); - _module.AddLabel(mergeLabel); - } - - index = target; - } - - return true; - } - - // The innermost structured loop's exit block and merge label, for its breaks; -1 outside loops. - private int _structuredLoopExit = -1; - private uint _structuredLoopMerge; - - // The last block in [header, end) that branches back to header, unconditionally or on a - // wave-uniform condition. - private bool TryFindLoopLatch(IReadOnlyList blocks, int header, int end, out int latch) - { - latch = -1; - var instructions = _request.Program.Instructions; - for (var index = end - 1; index >= header; index--) - { - var terminator = instructions[blocks[index].EndIndex - 1]; - if ((IsStructuredCondition(terminator.Opcode) || terminator.Opcode == "SBranch") && - TryResolveBranchBlock(blocks, terminator, out var target) && target == header) - { - latch = index; - return true; - } - } - - return false; - } - - // do { blocks [header, latch] } while (latch condition), bounded by the same step guard as - // the dispatcher so a guest loop that never ends cannot wedge the GPU queue. - private bool TryStructureLoop(IReadOnlyList blocks, int header, int latch, bool emit, out string error) - { - var outerExit = _structuredLoopExit; - var outerMerge = _structuredLoopMerge; - try - { - return TryStructureLoopCore(blocks, header, latch, emit, out error); - } - finally - { - _structuredLoopExit = outerExit; - _structuredLoopMerge = outerMerge; - } - } - - private bool TryStructureLoopCore(IReadOnlyList blocks, int header, int latch, bool emit, out string error) - { - _structuredLoopExit = latch + 1; - if (!emit) - { - _structuredLoopMerge = 0; - return TryStructureRange(blocks, header, latch + 1, latch + 1, emit: false, out error, latch); - } - - var headerLabel = _module.AllocateId(); - var bodyLabel = _module.AllocateId(); - var continueLabel = _module.AllocateId(); - var mergeLabel = _module.AllocateId(); - _structuredLoopMerge = mergeLabel; - _module.AddStatement(SpirvOp.Branch, headerLabel); - _module.AddLabel(headerLabel); - _module.AddStatement(SpirvOp.LoopMerge, mergeLabel, continueLabel, 0); - _module.AddStatement(SpirvOp.Branch, bodyLabel); - _module.AddLabel(bodyLabel); - if (!TryStructureRange(blocks, header, latch + 1, latch + 1, emit: true, out error, latch)) - { - return false; - } - - _module.AddStatement(SpirvOp.Branch, continueLabel); - _module.AddLabel(continueLabel); - var terminator = _request.Program.Instructions[blocks[latch].EndIndex - 1]; - var again = _module.ConstantBool(true); - if (terminator.Opcode != "SBranch") - { - TryGetBranchCondition(terminator.Opcode, out again); - } - - if (_maxDispatcherSteps > 0) - { - var steps = IAdd(Load(_uintType, _iterationGuard), UInt(1)); - Store(_iterationGuard, steps); - var withinLimit = _module.AddInstruction(SpirvOp.ULessThan, _boolType, steps, UInt((uint)_maxDispatcherSteps)); - again = _module.AddInstruction(SpirvOp.LogicalAnd, _boolType, again, withinLimit); - } - - _module.AddStatement(SpirvOp.BranchConditional, again, headerLabel, mergeLabel); - _module.AddLabel(mergeLabel); - return true; - } - - // SHARPEMU_STRUCTURED_FORWARD_BLOCKS=0 always emits the block dispatcher loop. - private static readonly bool StructuredForwardBlocks = !string.Equals( - Environment.GetEnvironmentVariable("SHARPEMU_STRUCTURED_FORWARD_BLOCKS"), "0", StringComparison.Ordinal); - - // The natural loops of the program: each block targeted by a backward branch heads a - // loop that ends at the last block branching back to it. Loops must nest; a program - // whose loops overlap keeps the block dispatcher. - private Dictionary _loopLatchByHeader = []; - - private bool TryBuildLoopRegions(IReadOnlyList blocks, out Dictionary latchByHeader) - { - latchByHeader = []; - for (var index = 0; index < blocks.Count; index++) - { - var terminator = _request.Program.Instructions[blocks[index].EndIndex - 1]; - if (terminator.Opcode != "SBranch" && !terminator.Opcode.StartsWith("SCbranch", StringComparison.Ordinal)) - { - continue; - } - - if (!TryGetBranchTargetPc(terminator, out var targetPc)) - { - return false; - } - - if (IsExitBranchTarget(_request.Program.Instructions, targetPc)) - { - continue; - } - - if (!TryFindBlock(blocks, targetPc, out var target)) - { - return false; - } - - if (target <= index) - { - latchByHeader[target] = latchByHeader.TryGetValue(target, out var latch) ? Math.Max(latch, index) : index; - } - } - - foreach (var (outerHeader, outerLatch) in latchByHeader) - { - foreach (var (innerHeader, innerLatch) in latchByHeader) - { - if (outerHeader < innerHeader && innerHeader <= outerLatch && innerLatch > outerLatch) - { - return false; - } - } - } - - return true; - } - - // Emits blocks first..last in order; a nested loop header opens a structured loop. - private bool TryEmitStructuredRange(IReadOnlyList blocks, int first, int last, int enclosingHeader, out string error) - { - error = string.Empty; - for (var index = first; index <= last;) - { - if (index != enclosingHeader && _loopLatchByHeader.TryGetValue(index, out var latch)) - { - if (!TryEmitStructuredLoop(blocks, index, latch, out error)) - { - return false; - } - - index = latch + 1; - continue; - } - - var isNext = _module.AddInstruction( - SpirvOp.LogicalAnd, - _boolType, - Load(_boolType, _programActive), - _module.AddInstruction(SpirvOp.IEqual, _boolType, Load(_uintType, _programCounter), UInt((uint)index))); - var blockBody = _module.AllocateId(); - var blockMerge = _module.AllocateId(); - _module.AddStatement(SpirvOp.SelectionMerge, blockMerge, 0); - _module.AddStatement(SpirvOp.BranchConditional, isNext, blockBody, blockMerge); - _module.AddLabel(blockBody); - if (!TryEmitBlock(blocks, index, out error)) - { - error = $"block=0x{blocks[index].StartPc:X}: {error}"; - return false; - } - - _module.AddStatement(SpirvOp.Branch, blockMerge); - _module.AddLabel(blockMerge); - index++; - } - - return true; - } - - // One natural loop: the body runs its blocks in order and repeats while the next block - // is the header. An exit leaves the program counter past the loop (or at an outer - // header), so the guards after the loop pick up where the invocation continues. - private bool TryEmitStructuredLoop(IReadOnlyList blocks, int header, int latch, out string error) - { - // Once a single-block loop is entered, its back edge already proves that - // this block is active and next. Keep its entry guard outside the loop; - // a conditional around every iteration obstructs driver optimization. - var singleBlock = header == latch; - var entryMerge = singleBlock ? _module.AllocateId() : 0u; - var skippedEntry = singleBlock ? _module.AllocateId() : 0u; - var loopHeader = _module.AllocateId(); - var loopBody = _module.AllocateId(); - var loopContinue = _module.AllocateId(); - var loopMerge = _module.AllocateId(); - if (singleBlock) - { - var enters = LogicalAnd(Load(_boolType, _programActive), - _module.AddInstruction(SpirvOp.IEqual, _boolType, Load(_uintType, _programCounter), UInt((uint)header))); - _module.AddStatement(SpirvOp.SelectionMerge, entryMerge, 0); - _module.AddStatement(SpirvOp.BranchConditional, enters, loopHeader, skippedEntry); - } - else - { - _module.AddStatement(SpirvOp.Branch, loopHeader); - } - _module.AddLabel(loopHeader); - _module.AddStatement(SpirvOp.LoopMerge, loopMerge, loopContinue, 0); - _module.AddStatement(SpirvOp.Branch, loopBody); - _module.AddLabel(loopBody); - if (!(singleBlock - ? TryEmitBlock(blocks, header, out error) - : TryEmitStructuredRange(blocks, header, latch, header, out error))) - { - return false; - } - - _module.AddStatement(SpirvOp.Branch, loopContinue); - _module.AddLabel(loopContinue); - var again = _module.AddInstruction( - SpirvOp.LogicalAnd, - _boolType, - Load(_boolType, _programActive), - _module.AddInstruction(SpirvOp.IEqual, _boolType, Load(_uintType, _programCounter), UInt((uint)header))); - if (_maxDispatcherSteps > 0) - { - // The dispatcher's safety valve, counted per loop iteration: a mistranslated - // exit ends the invocation instead of wedging the GPU queue. - var steps = IAdd(Load(_uintType, _iterationGuard), UInt(1)); - Store(_iterationGuard, steps); - var withinLimit = _module.AddInstruction(SpirvOp.ULessThan, _boolType, steps, UInt((uint)_maxDispatcherSteps)); - again = _module.AddInstruction(SpirvOp.LogicalAnd, _boolType, again, withinLimit); - } - - _module.AddStatement(SpirvOp.BranchConditional, again, loopHeader, loopMerge); - _module.AddLabel(loopMerge); - if (singleBlock) - { - _module.AddStatement(SpirvOp.Branch, entryMerge); - _module.AddLabel(skippedEntry); - // The previous guarded do-while still counted one iteration when - // the block was skipped. Preserve that diagnostic safety budget. - if (_maxDispatcherSteps > 0) - { - Store(_iterationGuard, IAdd(Load(_uintType, _iterationGuard), UInt(1))); - } - _module.AddStatement(SpirvOp.Branch, entryMerge); - _module.AddLabel(entryMerge); - } - return true; - } private static string FormatBlockStarts(IReadOnlyList blocks) { @@ -2146,14 +1983,21 @@ public static partial class Gen5SpirvTranslator { return true; } + if (instruction.Opcode == "SWaitcnt") + { + NoteWaitCount(instruction); + return true; + } + if (instruction.Opcode is "SNop" or "SSetregB32" or - "SWaitcnt" or "SInstPrefetch" or "STtraceData" or // Wave scheduling priority hint; no effect on results. "SSetprio" or + // Sleeps the wave for a number of clocks; no effect on results. + "SSleep" or // NGG shaders bracket their exports with s_sendmsg // (GS_ALLOC_REQ/DEALLOC) to reserve hardware export space; // exports are translated directly, so the message is moot. @@ -2166,11 +2010,17 @@ public static partial class Gen5SpirvTranslator { if (_stage == Gen5SpirvStage.Compute) { - // s_waitcnt vmcnt(0) + s_barrier also publishes buffer and image - // stores to the workgroup: AcquireRelease over uniform, workgroup - // and image memory. + // s_barrier orders execution; what it publishes is whatever the guest + // waited for first. s_waitcnt vmcnt(0) before it means buffer and image + // stores are being published to the workgroup, so the barrier carries + // AcquireRelease over uniform, workgroup and image memory. A barrier that + // only follows lgkmcnt(0) publishes LDS, and ordering device memory there + // costs a cache flush the guest never asked for: UE's FFT bloom + // convolution barriers 48 times per group, every one of them lgkmcnt-only. var workgroup = UInt(2); - var semantics = UInt(0x8 | 0x40 | 0x100 | 0x800); + var semantics = UInt(_vectorMemoryWaited == false + ? 0x8u | 0x100u + : 0x8u | 0x40u | 0x100u | 0x800u); _module.AddStatement( SpirvOp.ControlBarrier, workgroup, @@ -2692,19 +2542,60 @@ public static partial class Gen5SpirvTranslator // dword count, so this is a no-op for in-range compute addresses but // prevents out-of-bounds access when a graphics-stage scratch write // uses an arbitrary computed byte address. - var index = BitwiseAnd( - ShiftRightLogical(addressWithOffset, UInt(2)), - UInt(_ldsDwordMask)); - return _module.AddInstruction( + var dwordIndex = ShiftRightLogical(addressWithOffset, UInt(2)); + if (!_ldsIsWorkgroup) + { + var index = BitwiseAnd(dwordIndex, UInt(_ldsDwordMask)); + var privatePointer = _module.AddInstruction( + SpirvOp.AccessChain, + _ldsElementPointer, + _lds, + index); + // The private array has no initializer: zero a slot the first time this + // invocation addresses it, so it still reads zero where nothing was written. + var wordPointer = _module.AddInstruction( + SpirvOp.AccessChain, + _privateUintPointer, + _ldsWritten, + ShiftRightLogical(index, UInt(5))); + var bit = ShiftLeftLogical(UInt(1), BitwiseAnd(index, UInt(31))); + var word = Load(_uintType, wordPointer); + var unwritten = _module.AddInstruction( + SpirvOp.IEqual, _boolType, BitwiseAnd(word, bit), UInt(0)); + EmitConditional(unwritten, () => + { + Store(privatePointer, UInt(0)); + Store(wordPointer, BitwiseOr(word, bit)); + }); + return privatePointer; + } + + // Compute LDS follows the hardware: an address past the allocation reads 0 and + // drops writes. It must not wrap into the translator's dwords that follow the + // guest's (DeclareLds). Out-of-range accesses go to the dword that is always zero; + // StoreLds skips stores through them. + var inBounds = _module.AddInstruction( + SpirvOp.ULessThan, + _boolType, + dwordIndex, + UInt(_ldsGuestDwordLimit)); + var pointer = _module.AddInstruction( SpirvOp.AccessChain, _ldsElementPointer, _lds, - index); + SelectU(inBounds, dwordIndex, UInt(_ldsZeroDword))); + _ldsPointerInBounds[pointer] = inBounds; + return pointer; } private void StoreLds(uint pointer, uint value) { var active = Load(_boolType, _exec); + if (_ldsPointerInBounds.TryGetValue(pointer, out var inBounds)) + { + active = _module.AddInstruction(SpirvOp.LogicalAnd, _boolType, active, inBounds); + } + // Inactive lanes must not read and write back another lane's shared value. EmitConditional(active, () => Store(pointer, value)); } @@ -3027,7 +2918,7 @@ public static partial class Gen5SpirvTranslator return true; } - var vector = Load(_vec4Type, input); + var vector = LoadOrdinaryInterpolant(input); var component = _module.AddInstruction( SpirvOp.CompositeExtract, _floatType, @@ -3072,12 +2963,6 @@ public static partial class Gen5SpirvTranslator return true; } - if (control.Typed && instruction.Opcode.Contains("D16", StringComparison.Ordinal)) - { - error = $"unsupported buffer opcode {instruction.Opcode}"; - return false; - } - if (_stage == Gen5SpirvStage.Vertex && _vertexInputsByPc.TryGetValue(instruction.Pc, out var vertexInput)) { @@ -3144,7 +3029,7 @@ public static partial class Gen5SpirvTranslator } var specialized = info.Buffers[bindingIndex]; - var stride = UInt(specialized.PackedStride & 0x3FFF); + var stride = control.IndexEnabled ? BufferStride(bindingIndex, specialized.PackedStride) : UInt(0); var descriptorFormat = specialized.DescriptorFormat; var descriptorWord3 = UInt((specialized.DescriptorFormat << 12) | (specialized.DescriptorSwizzle & 0xFFF)); @@ -3302,8 +3187,7 @@ public static partial class Gen5SpirvTranslator return true; } - if (instruction.Opcode is "BufferStoreFormatX" or "BufferStoreFormatXy" or - "BufferStoreFormatXyz" or "BufferStoreFormatXyzw") + if (instruction.Opcode.StartsWith("BufferStoreFormat", StringComparison.Ordinal)) { if (descriptorFormat == 0) { @@ -3316,8 +3200,9 @@ public static partial class Gen5SpirvTranslator return false; } + var storeHighHalf = instruction.Opcode == "BufferStoreFormatD16HiX"; EmitExecConditional(() => TryEmitBufferFormatStore( - bindingIndex, byteAddress, control, descriptorWord3, descriptorFormat)); + bindingIndex, byteAddress, control, descriptorWord3, descriptorFormat, storeHighHalf)); return true; } @@ -3397,12 +3282,26 @@ public static partial class Gen5SpirvTranslator { if (!control.Typed) { + if (!_request.ForceGenericBufferFormats) + { + EmitStaticBufferFormatLoad( + bindingIndex, + byteAddress, + descriptorFormat, + specialized.DescriptorSwizzle & 0xFFF, + control.VectorData, + control.ComponentCount, + control.PackedD16); + return true; + } + EmitBufferFormatLoad( bindingIndex, byteAddress, descriptorWord3, control.VectorData, - control.DwordCount); + control.ComponentCount, + control.PackedD16); return true; } @@ -3438,20 +3337,12 @@ public static partial class Gen5SpirvTranslator error = string.Empty; if (!_request.BufferCandidateTableByMemoryIndex.TryGetValue(memoryIndex, out var table)) { - // A formatted load needs a statically known Vulkan view format. Most - // runtime V#s prove a bounded SRT candidate set above, but a few Yotei - // material-lookup paths merge descriptors through control flow and have - // no finite, safe candidate table. Do not reinterpret an arbitrary - // device address with a guessed format: a null read is the defined - // fallback, matching the bindless-image fallback in ResourceTracker. + // Untyped formatted loads carry their complete format and swizzle + // in the runtime V#. Decode those exact words through the page table + // when no bounded native candidate set exists. if (instruction.Opcode.StartsWith("BufferLoadFormat", StringComparison.Ordinal)) { - for (uint index = 0; index < control.DwordCount; index++) - { - StoreV(control.VectorData + index, UInt(0)); - } - - return true; + return TryEmitDeviceDescriptorBufferMemory(instruction, control, out error); } error = $"runtime buffer descriptor has no candidate table for {instruction.Opcode}"; @@ -3568,7 +3459,15 @@ public static partial class Gen5SpirvTranslator byteAddress, descriptorWord3, control.VectorData, - control.DwordCount); + control.ComponentCount, + control.PackedD16); + return true; + } + + if (instruction.Opcode.StartsWith("BufferStoreFormat", StringComparison.Ordinal)) + { + var storeHighHalf = instruction.Opcode == "BufferStoreFormatD16HiX"; + EmitExecConditional(() => EmitDeviceBufferFormatStore(baseAddress, size, byteAddress, descriptorWord3, control, storeHighHalf)); return true; } @@ -3681,6 +3580,8 @@ public static partial class Gen5SpirvTranslator var bothMapped = LogicalAnd(firstMapped, secondMapped); EmitConditional(bothMapped, () => { + NoteDeviceWrite(firstAddress); + NoteDeviceWrite(secondAddress); var originalLow = EmitAtomic( atomicOp, _uintType, DeviceWordPointer(firstPointer), 1, 0x48, () => LoadV(control.VectorData), () => UInt(0)); @@ -3709,6 +3610,7 @@ public static partial class Gen5SpirvTranslator var (pointer, mapped) = ResolveDeviceAddress(address); EmitConditional(mapped, () => { + NoteDeviceWrite(address); var original = EmitBufferFloatAtomic( DeviceWordPointer(pointer), LoadV(control.VectorData), @@ -3738,6 +3640,7 @@ public static partial class Gen5SpirvTranslator var (pointer, mapped) = ResolveDeviceAddress(address); EmitConditional(mapped, () => { + NoteDeviceWrite(address); var original = EmitAtomic( atomicOperation, _uintType, @@ -3754,12 +3657,171 @@ public static partial class Gen5SpirvTranslator return true; } + // The V# format is only known at run time, so each unified format the hardware + // defines gets its own branch, encoded exactly as a bound buffer's store. Only the + // bits of the transferred components are written. + private void EmitDeviceBufferFormatStore( + uint baseAddress, + uint size, + uint byteAddress, + uint descriptorWord3, + Gen5BufferMemoryControl control, + bool d16High = false) + { + // Each runtime format only encodes the element into the store scratch; the masked + // writes, with their page walks and atomic merges, are emitted once for all formats. + // Emitting them inside every format's branch multiplied them by the ~90 formats, and + // the driver's compile time grows with that many atomic loops (minutes per shader). + var unifiedFormat = BitwiseAnd(ShiftRightLogical(descriptorWord3, UInt(12)), UInt(0x7F)); + var elementAddress = And64(IAdd64(baseAddress, Widen(byteAddress)), ULong(DeviceAddressMask)); + for (var index = 0; index < _deviceStoreMaskScratch.Length; index++) + { + Store(_deviceStoreMaskScratch[index], UInt(0)); + } + + Store(_deviceStoreBytesScratch, UInt(0)); + var maxDwords = 0; + for (uint format = 1; format <= 0x7F; format++) + { + if (!Gfx10UnifiedFormat.TryDecode(format, out var dataFormat, out var numberFormat)) + { + continue; + } + + var componentCount = Math.Min(control.ComponentCount, Gfx10UnifiedFormat.ComponentCount(dataFormat)); + if (componentCount == 0) + { + continue; + } + + var elementBytes = Gfx10UnifiedFormat.GetAccessByteSize(dataFormat, componentCount); + var elementDwords = (int)((elementBytes + 3) / 4); + if (elementDwords > _deviceStoreMaskScratch.Length) + { + throw new InvalidOperationException($"A typed buffer element spans {elementDwords} dwords."); + } + + maxDwords = Math.Max(maxDwords, elementDwords); + EmitConditional(_module.AddInstruction(SpirvOp.IEqual, _boolType, unifiedFormat, UInt(format)), () => + { + var element = new (uint Value, uint Mask)[elementDwords]; + for (var index = 0; index < element.Length; index++) + { + element[index] = (UInt(0), 0); + } + + for (uint component = 0; component < componentCount; component++) + { + Gfx10UnifiedFormat.TryGetComponentLayout(dataFormat, component, out var byteOffset, out var bitOffset, out var bitCount); + var encoded = EncodeGfx10BufferComponent(LoadFormatStoreComponent(control, component, numberFormat, d16High), bitCount, numberFormat, dataFormat); + var dword = (int)(byteOffset / 4); + var bit = ((byteOffset & 3) * 8) + bitOffset; + var componentMask = bitCount == 32 ? uint.MaxValue : ((1u << (int)bitCount) - 1) << (int)bit; + var (value, mask) = element[dword]; + element[dword] = (BitwiseOr(value, bit == 0 ? encoded : ShiftLeftLogical(encoded, UInt(bit))), mask | componentMask); + } + + for (var index = 0; index < element.Length; index++) + { + Store(_deviceStoreValueScratch[index], BitwiseAnd(element[index].Value, UInt(element[index].Mask))); + Store(_deviceStoreMaskScratch[index], UInt(element[index].Mask)); + } + + Store(_deviceStoreBytesScratch, UInt(elementBytes)); + }); + } + + // An unknown format stores nothing: its byte count stays zero. + var bytes = Load(_uintType, _deviceStoreBytesScratch); + var allowed = LogicalAnd( + _module.AddInstruction(SpirvOp.INotEqual, _boolType, bytes, UInt(0)), + ULessThan64(IAdd64(Widen(byteAddress), Widen(_module.AddInstruction(SpirvOp.ISub, _uintType, bytes, UInt(1)))), size)); + for (var index = 0; index < maxDwords; index++) + { + StoreDeviceMaskedBits( + index == 0 ? elementAddress : IAdd64(elementAddress, ULong((ulong)index * sizeof(uint))), + Load(_uintType, _deviceStoreValueScratch[index]), + Load(_uintType, _deviceStoreMaskScratch[index]), + allowed); + } + } + + // Writes the masked bits of one element dword at a byte address that need not be + // dword aligned, with a mask known only at run time; `value` holds no bits outside it. + private void StoreDeviceMaskedBits(uint address64, uint value, uint maskValue, uint allowed) + { + var alignment = Narrow(And64(address64, ULong(3))); + var aligned = And64(address64, ULong(~3ul)); + var shift = ShiftLeftLogical(alignment, UInt(3)); + StoreDeviceMaskedWord(aligned, ShiftLeftLogical(value, shift), ShiftLeftLogical(maskValue, shift), allowed); + var unaligned = _module.AddInstruction(SpirvOp.INotEqual, _boolType, alignment, UInt(0)); + var carryShift = _module.AddInstruction(SpirvOp.ISub, _uintType, UInt(32), shift); + var carryValue = _module.AddInstruction(SpirvOp.Select, _uintType, unaligned, ShiftRightLogical(value, carryShift), UInt(0)); + var carryMask = _module.AddInstruction(SpirvOp.Select, _uintType, unaligned, ShiftRightLogical(maskValue, carryShift), UInt(0)); + StoreDeviceMaskedWord(IAdd64(aligned, ULong(4)), carryValue, carryMask, allowed); + } + + + private readonly Dictionary _deviceFormatLoadFunctions = new(); + private void EmitDeviceBufferFormatLoad( uint baseAddress, uint size, uint byteAddress, uint descriptorWord3, uint vectorData, + uint componentCount, + bool packedD16 = false) + { + if (!_deviceFormatLoadFunctions.TryGetValue(componentCount, out var function)) + { + function = _module.AllocateId(); + _deviceFormatLoadFunctions.Add(componentCount, function); + } + + var values = _module.AddInstruction(SpirvOp.FunctionCall, _uvec4Type, function, + baseAddress, size, byteAddress, descriptorWord3, UInt(_deviceAddressInstructionPc)); + var format = BitwiseAnd(ShiftRightLogical(descriptorWord3, UInt(12)), UInt(0x7F)); + var (_, numberFormat) = DecodeGfx10BufferFormat(format); + for (uint component = 0; component < componentCount; component++) + { + var value = _module.AddInstruction(SpirvOp.CompositeExtract, _uintType, values, component); + StoreFormattedBufferComponent(vectorData, component, value, numberFormat, packedD16); + } + } + + private void EmitDeviceFormatLoadFunctions() + { + foreach (var (components, function) in _deviceFormatLoadFunctions) + { + var type = _module.TypeFunction(_uvec4Type, + _ulongType, _ulongType, _uintType, _uintType, _uintType); + // Keeping the shared decoder as a function avoids expanding the + // complete runtime format path at every read site in the driver. + _module.BeginFunction(_uvec4Type, type, function, SpirvFunctionControl.DontInline); + _module.AddName(function, $"loadDeviceBufferFormat{components}"); + var address = _module.AddFunctionParameter(_ulongType); + var size = _module.AddFunctionParameter(_ulongType); + var offset = _module.AddFunctionParameter(_uintType); + var descriptor = _module.AddFunctionParameter(_uintType); + _deviceAddressInstructionPcValue = _module.AddFunctionParameter(_uintType); + _module.AddLabel(); + var values = LoadDeviceBufferFormatValues(address, size, offset, descriptor, components); + _deviceAddressInstructionPcValue = null; + var result = new uint[4]; + for (var component = 0; component < result.Length; component++) + result[component] = component < values.Length ? values[component] : UInt(0); + _module.AddStatement(SpirvOp.ReturnValue, + _module.AddInstruction(SpirvOp.CompositeConstruct, _uvec4Type, result)); + _module.EndFunction(); + } + } + + private uint[] LoadDeviceBufferFormatValues( + uint baseAddress, + uint size, + uint byteAddress, + uint descriptorWord3, uint componentCount) { var unifiedFormat = BitwiseAnd( @@ -3767,17 +3829,52 @@ public static partial class Gen5SpirvTranslator UInt(0x7F)); var (dataFormat, numberFormat) = DecodeGfx10BufferFormat(unifiedFormat); + // The element spans at most 16 bytes; read its aligned dwords once and take + // every component from them instead of reading memory per component byte. + var window = new uint[5]; + var alignedStart = BitwiseAnd(byteAddress, UInt(~3u)); + for (var index = 0; index < window.Length; index++) + { + window[index] = LoadDeviceBufferWindowWord(baseAddress, size, index == 0 ? alignedStart : IAdd(alignedStart, UInt((uint)index * sizeof(uint)))); + } + + // For a partial vector, select its source layout before conversion. A + // dynamic swizzle does not require converting the unrequested channels. + if (componentCount < 4) + { + var values = new uint[componentCount]; + var constants = new uint[componentCount]; + var allInBounds = _module.ConstantBool(true); + var oneValue = Gfx10FormatOne(numberFormat); + for (uint destination = 0; destination < componentCount; destination++) + { + var selector = BitwiseAnd(ShiftRightLogical(descriptorWord3, UInt(destination * 3)), UInt(7)); + var component = BitwiseAnd(selector, UInt(3)); + var isMemory = _module.AddInstruction(SpirvOp.UGreaterThanEqual, _boolType, selector, UInt(4)); + var value = LoadGfx10DeviceBufferFormatComponent(window, size, byteAddress, + dataFormat, numberFormat, component, out var selectedInBounds); + constants[destination] = SelectUInt(selector, 1, oneValue, UInt(0)); + values[destination] = _module.AddInstruction(SpirvOp.Select, _uintType, + isMemory, value, constants[destination]); + allInBounds = LogicalAnd(allInBounds, _module.AddInstruction(SpirvOp.Select, + _boolType, isMemory, selectedInBounds, _module.ConstantBool(true))); + } + for (uint destination = 0; destination < componentCount; destination++) + values[destination] = _module.AddInstruction(SpirvOp.Select, + _uintType, allInBounds, values[destination], constants[destination]); + return values; + } var canonical = new uint[4]; var componentBounds = new uint[4]; for (var component = 0; component < canonical.Length; component++) { canonical[component] = LoadGfx10DeviceBufferFormatComponent( - baseAddress, + window, size, byteAddress, dataFormat, numberFormat, - component, + UInt((uint)component), out componentBounds[component]); } @@ -3803,6 +3900,7 @@ public static partial class Gen5SpirvTranslator inBounds = _module.AddInstruction(SpirvOp.LogicalAnd, _boolType, inBounds, selectedInBounds); } + var valuesFull = new uint[componentCount]; var one = Gfx10FormatOne(numberFormat); for (uint destination = 0; destination < componentCount; destination++) { @@ -3813,38 +3911,22 @@ public static partial class Gen5SpirvTranslator value = SelectUInt(selector, 5, canonical[1], value); value = SelectUInt(selector, 6, canonical[2], value); value = SelectUInt(selector, 7, canonical[3], value); - StoreV( - vectorData + destination, - _module.AddInstruction(SpirvOp.Select, _uintType, inBounds, value, constant)); + valuesFull[destination] = _module.AddInstruction(SpirvOp.Select, _uintType, inBounds, value, constant); } + return valuesFull; } private uint LoadGfx10DeviceBufferFormatComponent( - uint baseAddress, + uint[] window, uint size, uint elementAddress, uint dataFormat, uint numberFormat, - int component, + uint component, out uint componentInBounds) { - var byteOffset = UInt(0); - var bitOffset = UInt(0); - var bitCount = UInt(0); + var (byteOffset, bitOffset, bitCount) = DecodeGfx10ComponentLayout(dataFormat, component); - void SetLayout(uint format, uint bytes, uint bits, uint count) - { - var matches = _module.AddInstruction(SpirvOp.IEqual, _boolType, dataFormat, UInt(format)); - byteOffset = _module.AddInstruction(SpirvOp.Select, _uintType, matches, UInt(bytes), byteOffset); - bitOffset = _module.AddInstruction(SpirvOp.Select, _uintType, matches, UInt(bits), bitOffset); - bitCount = _module.AddInstruction(SpirvOp.Select, _uintType, matches, UInt(count), bitCount); - } - - foreach (var layout in Gfx10UnifiedFormat.ComponentLayouts) - { - if (layout.Component == (uint)component) - SetLayout(layout.DataFormat, layout.ByteOffset, layout.BitOffset, layout.BitCount); - } var componentAddress = IAdd(elementAddress, byteOffset); var componentBytes = ShiftRightLogical(IAdd(IAdd(bitOffset, bitCount), UInt(7)), UInt(3)); @@ -3857,16 +3939,26 @@ public static partial class Gen5SpirvTranslator _boolType, LogicalNot(hasComponent), inRange); - var packed = UInt(0); - for (uint index = 0; index < sizeof(uint); index++) + // Byte offset of the component inside the window, then a funnel shift of the + // two dwords it straddles. + var relative = IAdd(BitwiseAnd(elementAddress, UInt(3)), byteOffset); + var wordIndex = ShiftRightLogical(relative, UInt(2)); + var shift = ShiftLeftLogical(BitwiseAnd(relative, UInt(3)), UInt(3)); + var low = window[0]; + var high = window[1]; + for (uint index = 1; index < 4; index++) { - var address = index == 0 ? componentAddress : IAdd(componentAddress, UInt(index)); - var word = LoadDeviceBufferWord(baseAddress, size, address, accessAllowed); - var shift = ShiftLeftLogical(BitwiseAnd(address, UInt(3)), UInt(3)); - var value = BitwiseAnd(ShiftRightLogical(word, shift), UInt(0xFF)); - packed = BitwiseOr(packed, ShiftLeftLogical(value, UInt(index * 8))); + var selected = _module.AddInstruction(SpirvOp.IEqual, _boolType, wordIndex, UInt(index)); + low = _module.AddInstruction(SpirvOp.Select, _uintType, selected, window[index], low); + high = _module.AddInstruction(SpirvOp.Select, _uintType, selected, window[index + 1], high); } + var carry = _module.AddInstruction(SpirvOp.Select, _uintType, + _module.AddInstruction(SpirvOp.IEqual, _boolType, shift, UInt(0)), UInt(0), + ShiftLeftLogical(high, _module.AddInstruction(SpirvOp.ISub, _uintType, UInt(32), shift))); + var packed = _module.AddInstruction(SpirvOp.Select, _uintType, accessAllowed, + BitwiseOr(ShiftRightLogical(low, shift), carry), UInt(0)); + var raw = _module.AddInstruction(SpirvOp.BitFieldUExtract, _uintType, packed, bitOffset, bitCount); var converted = ConvertGfx10BufferComponent(raw, bitCount, numberFormat, dataFormat); var valid = _module.AddInstruction(SpirvOp.INotEqual, _boolType, bitCount, UInt(0)); @@ -3875,7 +3967,7 @@ public static partial class Gen5SpirvTranslator _uintType, valid, converted, - component == 3 ? Gfx10FormatOne(numberFormat) : UInt(0)); + SelectUInt(component, 3, Gfx10FormatOne(numberFormat), UInt(0))); } private void EmitBufferFormatLoad( @@ -3883,27 +3975,94 @@ public static partial class Gen5SpirvTranslator uint byteAddress, uint descriptorWord3, uint vectorData, - uint componentCount) + uint componentCount, + bool packedD16 = false) { var unifiedFormat = BitwiseAnd( ShiftRightLogical(descriptorWord3, UInt(12)), UInt(0x7F)); var (dataFormat, numberFormat) = DecodeGfx10BufferFormat(unifiedFormat); + var one = Gfx10FormatOne(numberFormat); - var canonical = new uint[4]; - var componentBounds = new uint[4]; - for (var component = 0; component < canonical.Length; component++) - { - canonical[component] = LoadGfx10BufferFormatComponent( - bindingIndex, - byteAddress, - dataFormat, - numberFormat, - component, - out componentBounds[component]); - } + // Dword-aligned 32-bit components whose number format keeps the raw bits (most + // vertex streams) load one word per component; every other format takes the + // generic byte-wise decode. The descriptor is uniform, so the branch never diverges. + var wordCount = UInt(0); + wordCount = SelectUInt(dataFormat, 4, UInt(1), wordCount); + wordCount = SelectUInt(dataFormat, 11, UInt(2), wordCount); + wordCount = SelectUInt(dataFormat, 13, UInt(3), wordCount); + wordCount = SelectUInt(dataFormat, 14, UInt(4), wordCount); + var rawBits = _module.AddInstruction( + SpirvOp.LogicalOr, _boolType, + _module.AddInstruction(SpirvOp.IEqual, _boolType, numberFormat, UInt(7)), + _module.AddInstruction( + SpirvOp.LogicalOr, _boolType, + _module.AddInstruction(SpirvOp.IEqual, _boolType, numberFormat, UInt(4)), + _module.AddInstruction(SpirvOp.IEqual, _boolType, numberFormat, UInt(5)))); + var wordPath = _module.AddInstruction( + SpirvOp.LogicalAnd, _boolType, + _module.AddInstruction( + SpirvOp.LogicalAnd, _boolType, + _module.AddInstruction(SpirvOp.INotEqual, _boolType, wordCount, UInt(0)), + rawBits), + _module.AddInstruction( + SpirvOp.IEqual, _boolType, BitwiseAnd(byteAddress, UInt(3)), UInt(0))); - // Only selected memory components contribute to the shared bounds check. + EmitConditional( + wordPath, + () => + { + var dwordAddress = ShiftRightLogical(byteAddress, UInt(2)); + var canonical = new uint[4]; + var componentBounds = new uint[4]; + for (uint component = 0; component < 4; component++) + { + var address = component == 0 ? dwordAddress : IAdd(dwordAddress, UInt(component)); + var present = _module.AddInstruction( + SpirvOp.ULessThan, _boolType, UInt(component), wordCount); + canonical[component] = _module.AddInstruction( + SpirvOp.Select, _uintType, present, + LoadBufferWord(bindingIndex, address), + component == 3 ? one : UInt(0)); + componentBounds[component] = _module.AddInstruction( + SpirvOp.LogicalOr, _boolType, + _module.AddInstruction(SpirvOp.LogicalNot, _boolType, present), + IsBufferWordInRange(bindingIndex, address)); + } + + StoreBufferFormatComponents(descriptorWord3, vectorData, componentCount, one, canonical, componentBounds, numberFormat, packedD16); + }, + () => + { + var canonical = new uint[4]; + var componentBounds = new uint[4]; + for (var component = 0; component < canonical.Length; component++) + { + canonical[component] = LoadGfx10BufferFormatComponent( + bindingIndex, + byteAddress, + dataFormat, + numberFormat, + component, + out componentBounds[component]); + } + + StoreBufferFormatComponents(descriptorWord3, vectorData, componentCount, one, canonical, componentBounds, numberFormat, packedD16); + }); + } + + // Applies the descriptor swizzle to the memory components; only selected memory + // components contribute to the shared bounds check. + private void StoreBufferFormatComponents( + uint descriptorWord3, + uint vectorData, + uint componentCount, + uint one, + uint[] canonical, + uint[] componentBounds, + uint numberFormat, + bool packedD16) + { var selectors = new uint[componentCount]; var inBounds = _module.ConstantBool(true); for (uint destination = 0; destination < componentCount; destination++) @@ -3923,7 +4082,6 @@ public static partial class Gen5SpirvTranslator inBounds = _module.AddInstruction(SpirvOp.LogicalAnd, _boolType, inBounds, selectedInBounds); } - var one = Gfx10FormatOne(numberFormat); for (uint destination = 0; destination < componentCount; destination++) { var selector = selectors[destination]; @@ -3933,15 +4091,163 @@ public static partial class Gen5SpirvTranslator value = SelectUInt(selector, 5, canonical[1], value); value = SelectUInt(selector, 6, canonical[2], value); value = SelectUInt(selector, 7, canonical[3], value); - StoreV( - vectorData + destination, - _module.AddInstruction(SpirvOp.Select, _uintType, inBounds, value, constant)); + StoreFormattedBufferComponent(vectorData, destination, + _module.AddInstruction(SpirvOp.Select, _uintType, inBounds, value, constant), numberFormat, packedD16); } } - // Check the first and last dwords of the required range together. + // EmitBufferFormatLoad for a descriptor format and swizzle known when the shader is + // translated: only that format's layout and conversion are emitted, and only the + // memory components the swizzle selects are read. Same results, bit for bit. + private void EmitStaticBufferFormatLoad( + int bindingIndex, + uint byteAddress, + uint unifiedFormat, + uint swizzle, + uint vectorData, + uint componentCount, + bool packedD16 = false) + { + Gfx10UnifiedFormat.TryDecode(unifiedFormat & 0x7F, out var dataFormat, out var numberFormat); + var one = UInt(numberFormat is 4 or 5 ? 1u : 0x3F800000u); + var canonical = new uint[4]; + var componentBounds = new uint[4]; + var inBounds = _module.ConstantBool(true); + for (uint destination = 0; destination < componentCount; destination++) + { + var selector = (swizzle >> (int)(destination * 3)) & 7; + if (selector < 4) continue; + var component = selector - 4; + if (canonical[component] == 0) + { + canonical[component] = LoadStaticBufferFormatComponent( + bindingIndex, byteAddress, dataFormat, numberFormat, component, out componentBounds[component]); + } + + inBounds = LogicalAnd(inBounds, componentBounds[component]); + } + + for (uint destination = 0; destination < componentCount; destination++) + { + var selector = (swizzle >> (int)(destination * 3)) & 7; + var constant = selector == 1 ? one : UInt(0); + var value = selector >= 4 + ? _module.AddInstruction(SpirvOp.Select, _uintType, inBounds, canonical[selector - 4], constant) + : constant; + StoreFormattedBufferComponent(vectorData, destination, value, UInt(numberFormat), packedD16); + } + } + + private void StoreFormattedBufferComponent(uint vectorData, uint component, uint value, uint numberFormat, bool packedD16) + { + if (!packedD16) + { + StoreV(vectorData + component, value); + return; + } + var integer = _module.AddInstruction(SpirvOp.LogicalOr, _boolType, + _module.AddInstruction(SpirvOp.IEqual, _boolType, numberFormat, UInt(4)), + _module.AddInstruction(SpirvOp.IEqual, _boolType, numberFormat, UInt(5))); + var pair = _module.AddInstruction(SpirvOp.CompositeConstruct, _vec2Type, + Bitcast(_floatType, value), Float(0)); + var half = _module.AddInstruction(SpirvOp.Select, _uintType, integer, + BitwiseAnd(value, UInt(0xFFFF)), BitwiseAnd(Ext(58, _uintType, pair), UInt(0xFFFF))); + var register = vectorData + component / 2; + StoreV(register, (component & 1) == 0 ? half + : BitwiseOr(BitwiseAnd(LoadV(register), UInt(0xFFFF)), ShiftLeftLogical(half, UInt(16)))); + } + + // LoadGfx10BufferFormatComponent for a known data and number format. + private uint LoadStaticBufferFormatComponent( + int bindingIndex, + uint elementAddress, + uint dataFormat, + uint numberFormat, + uint component, + out uint componentInBounds) + { + if (!Gfx10UnifiedFormat.TryGetComponentLayout(dataFormat, component, out var byteOffset, out var bitOffset, out var bitCount) || + bitCount == 0) + { + // A component the format lacks: W reads as one, the others as zero. + componentInBounds = _module.ConstantBool(true); + return component == 3 ? UInt(numberFormat is 4 or 5 ? 1u : 0x3F800000u) : UInt(0); + } + + var componentAddress = byteOffset == 0 ? elementAddress : IAdd(elementAddress, UInt(byteOffset)); + var packed = LoadUnalignedBufferWord(bindingIndex, componentAddress); + componentInBounds = IsBufferElementInRange(bindingIndex, componentAddress, UInt(((bitOffset + bitCount + 7) >> 3) - 1)); + var raw = bitCount == 32 && bitOffset == 0 + ? packed + : _module.AddInstruction(SpirvOp.BitFieldUExtract, _uintType, packed, UInt(bitOffset), UInt(bitCount)); + return ConvertStaticBufferComponent(raw, bitCount, numberFormat, dataFormat); + } + + // ConvertGfx10BufferComponent for a known format: the same operations, only the one + // the number format selects. + private uint ConvertStaticBufferComponent(uint raw, uint bitCount, uint numberFormat, uint dataFormat) + { + var lowMask = bitCount == 32 ? uint.MaxValue : (1u << (int)bitCount) - 1; + uint SignedRaw() => _module.AddInstruction( + SpirvOp.BitFieldSExtract, _intType, Bitcast(_intType, raw), UInt(0), UInt(bitCount)); + switch (numberFormat) + { + case 0: + return Bitcast(_uintType, _module.AddInstruction( + SpirvOp.FDiv, _floatType, + _module.AddInstruction(SpirvOp.ConvertUToF, _floatType, raw), + _module.AddInstruction(SpirvOp.ConvertUToF, _floatType, UInt(lowMask)))); + case 1: + { + var snorm = _module.AddInstruction( + SpirvOp.FDiv, _floatType, + _module.AddInstruction(SpirvOp.ConvertSToF, _floatType, SignedRaw()), + _module.AddInstruction(SpirvOp.ConvertUToF, _floatType, UInt(lowMask >> 1))); + snorm = _module.AddInstruction( + SpirvOp.Select, _floatType, + _module.AddInstruction(SpirvOp.FOrdLessThan, _boolType, snorm, Float(-1f)), + Float(-1f), + snorm); + return Bitcast(_uintType, snorm); + } + case 2: + return Bitcast(_uintType, _module.AddInstruction(SpirvOp.ConvertUToF, _floatType, raw)); + case 3: + return Bitcast(_uintType, _module.AddInstruction(SpirvOp.ConvertSToF, _floatType, SignedRaw())); + case 5: + return Bitcast(_uintType, SignedRaw()); + case 7: + if (dataFormat is 6 or 7) + { + return DecodeUnsignedMiniFloat(raw, UInt(bitCount)); + } + + if (bitCount == 16) + { + var unpacked = Ext(62, _vec2Type, BitwiseAnd(raw, UInt(0xFFFF))); + return Bitcast(_uintType, _module.AddInstruction(SpirvOp.CompositeExtract, _floatType, unpacked, 0)); + } + + return raw; + default: + return raw; + } + } + + // A native view can include a partial final word. Check guest byte bounds + // before extracting its components; padding is not guest data. private uint IsBufferElementInRange(int bindingIndex, uint byteAddress, uint lastByteOffset) { + if (BufferMayHaveTailPadding(bindingIndex)) + { + var end = IAdd(byteAddress, lastByteOffset); + var length = BufferByteLength(bindingIndex); + return LogicalAnd( + _module.AddInstruction(SpirvOp.ULessThan, _boolType, byteAddress, length), + LogicalAnd( + _module.AddInstruction(SpirvOp.ULessThan, _boolType, end, length), + _module.AddInstruction(SpirvOp.UGreaterThanEqual, _boolType, end, byteAddress))); + } var first = IsBufferWordInRange(bindingIndex, ShiftRightLogical(byteAddress, UInt(2))); var last = IsBufferWordInRange( bindingIndex, @@ -3976,23 +4282,23 @@ public static partial class Gen5SpirvTranslator IsBufferElementInRange( bindingIndex, byteAddress, - UInt(Gfx10UnifiedFormat.GetAccessByteSize(dataFormat, control.DwordCount) - 1))); + UInt(Gfx10UnifiedFormat.GetAccessByteSize(dataFormat, control.ComponentCount) - 1))); var dataFormatId = UInt(dataFormat); var numberFormatId = UInt(numberFormat); - for (uint destination = 0; destination < control.DwordCount; destination++) + for (uint destination = 0; destination < control.ComponentCount; destination++) { - var value = destination < componentCount - ? LoadGfx10BufferFormatComponent( - bindingIndex, - byteAddress, - dataFormatId, - numberFormatId, - (int)destination, - out _) - : UInt(0); - StoreV( - control.VectorData + destination, - _module.AddInstruction(SpirvOp.Select, _uintType, valid, value, UInt(0))); + var value = destination >= componentCount ? UInt(0) + : _request.ForceGenericBufferFormats + ? LoadGfx10BufferFormatComponent( + bindingIndex, + byteAddress, + dataFormatId, + numberFormatId, + (int)destination, + out _) + : LoadStaticBufferFormatComponent(bindingIndex, byteAddress, dataFormat, numberFormat, destination, out _); + StoreFormattedBufferComponent(control.VectorData, destination, + _module.AddInstruction(SpirvOp.Select, _uintType, valid, value, UInt(0)), numberFormatId, control.PackedD16); } return true; @@ -4005,14 +4311,15 @@ public static partial class Gen5SpirvTranslator uint byteAddress, Gen5BufferMemoryControl control, uint descriptorWord3, - uint unifiedFormat) + uint unifiedFormat, + bool d16High = false) { if (!Gfx10UnifiedFormat.TryDecode(unifiedFormat, out var dataFormat, out var numberFormat)) { return false; } - var componentCount = Math.Min(control.DwordCount, Gfx10UnifiedFormat.ComponentCount(dataFormat)); + var componentCount = Math.Min(control.ComponentCount, Gfx10UnifiedFormat.ComponentCount(dataFormat)); if (componentCount == 0) { return false; @@ -4036,7 +4343,7 @@ public static partial class Gen5SpirvTranslator StoreBufferWord( bindingIndex, byteOffset == 0 ? dwordAddress : IAdd(dwordAddress, UInt(byteOffset / 4)), - LoadV(control.VectorData + component)); + LoadFormatStoreComponent(control, component, numberFormat, d16High)); } return; @@ -4052,7 +4359,7 @@ public static partial class Gen5SpirvTranslator { Gfx10UnifiedFormat.TryGetComponentLayout(dataFormat, component, out var byteOffset, out var bitOffset, out var bitCount); var encoded = EncodeGfx10BufferComponent( - LoadV(control.VectorData + component), + LoadFormatStoreComponent(control, component, numberFormat, d16High), bitCount, numberFormat, dataFormat); @@ -4069,6 +4376,31 @@ public static partial class Gen5SpirvTranslator return true; } + // A D16_HI store takes its component from the register's high half: a half float + // for float and normalized formats, a 16-bit integer otherwise. + private uint LoadFormatStoreComponent(Gen5BufferMemoryControl control, uint component, uint numberFormat, bool d16High) + { + var register = control.VectorData + (control.PackedD16 ? component / 2 : component); + return StoreFormatRegister(LoadV(register), numberFormat, + control.PackedD16 ? (component & 1) != 0 : d16High, control.PackedD16); + } + + private uint StoreFormatRegister(uint register, uint numberFormat, bool d16High, bool packedD16 = false) + { + if (!d16High && !packedD16) + { + return register; + } + + var half = d16High ? ShiftRightLogical(register, UInt(16)) : BitwiseAnd(register, UInt(0xFFFF)); + return numberFormat switch + { + 4 => half, + 5 => Bitcast(_uintType, _module.AddInstruction(SpirvOp.BitFieldSExtract, _intType, Bitcast(_intType, half), UInt(0), UInt(16))), + _ => Bitcast(_uintType, _module.AddInstruction(SpirvOp.CompositeExtract, _floatType, Ext(62, _vec2Type, half), 0u)), + }; + } + // Converts one register to the bits its component stores, per the number format. private uint EncodeGfx10BufferComponent(uint value, uint bitCount, uint numberFormat, uint dataFormat) { @@ -4284,6 +4616,38 @@ public static partial class Gen5SpirvTranslator UInt(0xFF))); } + private uint _gfx10ComponentLayoutTable; + + // Decoded GFX10 data formats occupy four bits. Keep component selection dynamic + // without expanding the entire format catalogue at every load instruction. + private (uint ByteOffset, uint BitOffset, uint BitCount) DecodeGfx10ComponentLayout(uint dataFormat, int component) + => DecodeGfx10ComponentLayout(dataFormat, UInt((uint)component)); + + private (uint ByteOffset, uint BitOffset, uint BitCount) DecodeGfx10ComponentLayout(uint dataFormat, uint component) + { + if (_gfx10ComponentLayoutTable == 0) + { + var values = new uint[16 * 4]; + foreach (var layout in Gfx10UnifiedFormat.ComponentLayouts) + values[layout.DataFormat * 4 + layout.Component] = + layout.ByteOffset | (layout.BitOffset << 8) | (layout.BitCount << 16); + var entries = values.Select(UInt).ToArray(); + var type = _module.TypeArray(_uintType, (uint)entries.Length); + _gfx10ComponentLayoutTable = _module.AddGlobalVariable( + _module.TypePointer(SpirvStorageClass.Private, type), SpirvStorageClass.Private, + _module.ConstantComposite(type, entries)); + _module.AddName(_gfx10ComponentLayoutTable, "gfx10ComponentLayouts"); + _interfaces.Add(_gfx10ComponentLayoutTable); + } + var index = IAdd(ShiftLeftLogical(dataFormat, UInt(2)), component); + var pointer = _module.AddInstruction(SpirvOp.AccessChain, _privateUintPointer, + _gfx10ComponentLayoutTable, index); + var entry = Load(_uintType, pointer); + return (BitwiseAnd(entry, UInt(255)), + BitwiseAnd(ShiftRightLogical(entry, UInt(8)), UInt(255)), + ShiftRightLogical(entry, UInt(16))); + } + private uint LoadGfx10BufferFormatComponent( int bindingIndex, uint elementAddress, @@ -4292,45 +4656,8 @@ public static partial class Gen5SpirvTranslator int component, out uint componentInBounds) { - var byteOffset = UInt(0); - var bitOffset = UInt(0); - var bitCount = UInt(0); + var (byteOffset, bitOffset, bitCount) = DecodeGfx10ComponentLayout(dataFormat, component); - void SetLayout(uint format, uint bytes, uint bits, uint count) - { - var matches = _module.AddInstruction( - SpirvOp.IEqual, - _boolType, - dataFormat, - UInt(format)); - byteOffset = _module.AddInstruction( - SpirvOp.Select, - _uintType, - matches, - UInt(bytes), - byteOffset); - bitOffset = _module.AddInstruction( - SpirvOp.Select, - _uintType, - matches, - UInt(bits), - bitOffset); - bitCount = _module.AddInstruction( - SpirvOp.Select, - _uintType, - matches, - UInt(count), - bitCount); - } - - // The layout is selected at run time from the descriptor's data format. - foreach (var layout in Gfx10UnifiedFormat.ComponentLayouts) - { - if (layout.Component == (uint)component) - { - SetLayout(layout.DataFormat, layout.ByteOffset, layout.BitOffset, layout.BitCount); - } - } var packed = LoadUnalignedBufferWord( bindingIndex, @@ -4597,23 +4924,27 @@ public static partial class Gen5SpirvTranslator whenTrue, whenFalse); + // The four bytes at a byte address span at most two words, so one funnel shift of those + // two words is the whole value. Reading each byte from its own bounds-checked word load + // (four loads, four range tests and a byte assembly) costs twice as much for the same + // result: a word outside the buffer reads zero either way, so the bytes it would have + // contributed stay zero, and the formatted-load fallback does this per component. private uint LoadUnalignedBufferWord(int bindingIndex, uint byteAddress) { - var result = UInt(0); - for (uint index = 0; index < 4; index++) - { - var address = index == 0 - ? byteAddress - : IAdd(byteAddress, UInt(index)); - var dwordAddress = ShiftRightLogical(address, UInt(2)); - var bitOffset = ShiftLeftLogical(BitwiseAnd(address, UInt(3)), UInt(3)); - var value = BitwiseAnd( - ShiftRightLogical(LoadBufferWord(bindingIndex, dwordAddress), bitOffset), - UInt(0xFF)); - result = BitwiseOr(result, ShiftLeftLogical(value, UInt(index * 8))); - } - - return result; + var dwordAddress = ShiftRightLogical(byteAddress, UInt(2)); + var bitOffset = ShiftLeftLogical(BitwiseAnd(byteAddress, UInt(3)), UInt(3)); + var low = LoadBufferWord(bindingIndex, dwordAddress); + var high = LoadBufferWord(bindingIndex, IAdd(dwordAddress, UInt(1))); + // A shift by the word width is undefined, so the aligned case keeps the low word. + var spanning = BitwiseOr( + ShiftRightLogical(low, bitOffset), + ShiftLeftLogical(high, _module.AddInstruction(SpirvOp.ISub, _uintType, UInt(32), bitOffset))); + return _module.AddInstruction( + SpirvOp.Select, + _uintType, + IsNotZero(bitOffset), + spanning, + low); } private uint LoadSubdwordBufferValue( @@ -4629,6 +4960,11 @@ public static partial class Gen5SpirvTranslator var raw = BitwiseAnd( LoadUnalignedBufferWord(bindingIndex, byteAddress), UInt(byteCount == 1 ? 0xFFu : 0xFFFFu)); + if (BufferMayHaveTailPadding(bindingIndex)) + { + raw = _module.AddInstruction(SpirvOp.Select, _uintType, + IsBufferElementInRange(bindingIndex, byteAddress, UInt(byteCount - 1)), raw, UInt(0)); + } if (signExtend) { raw = Bitcast( @@ -4670,10 +5006,16 @@ public static partial class Gen5SpirvTranslator value = ShiftRightLogical(value, UInt(sourceShift)); } - StoreBufferElementBits( - bindingIndex, - byteAddress, - [(value, byteCount == 1 ? 0xFFu : 0xFFFFu)]); + void StoreBytes() => StoreBufferElementBits( + bindingIndex, byteAddress, [(value, byteCount == 1 ? 0xFFu : 0xFFFFu)]); + if (BufferMayHaveTailPadding(bindingIndex)) + { + EmitConditional(IsBufferElementInRange(bindingIndex, byteAddress, UInt(byteCount - 1)), StoreBytes); + } + else + { + StoreBytes(); + } } // Stores an element of up to four dwords at any byte alignment. Each touched @@ -4745,11 +5087,8 @@ public static partial class Gen5SpirvTranslator EmitConditional( full, () => Store(pointer, value), - () => EmitAtomicWordUpdate( - pointer, - observed => BitwiseOr( - BitwiseAnd(observed, _module.AddInstruction(SpirvOp.Not, _uintType, mask)), - value))); + () => EmitAtomicWordUpdate(pointer, observed => BitwiseOr( + BitwiseAnd(observed, _module.AddInstruction(SpirvOp.Not, _uintType, mask)), value))); }); } @@ -4895,6 +5234,39 @@ public static partial class Gen5SpirvTranslator return true; } + // Candidates in one descriptor class, the first anywhere and the rest in consecutive + // elements, are read by index; the class keeps their per-candidate typing identical. + private bool TryGetDynamicImageElement( + uint selector, + IReadOnlyList<(uint Resource, uint Element)> elements, + out uint dynamicElement, + out (uint Resource, uint Element) representative) + { + dynamicElement = uint.MaxValue; + representative = default; + if (!_request.SupportsNonUniformImageIndexing || elements.Count < 3) + return false; + + var info = _request.Resources.Info; + var kind = ImageDescriptorBinding.ForImage(info.Images[(int)elements[1].Resource]); + for (var index = 0; index < elements.Count; index++) + { + var image = info.Images[(int)elements[index].Resource]; + if (ImageDescriptorBinding.ForImage(image) != kind || + (index >= 1 && elements[index].Element != elements[1].Element + (uint)(index - 1)) || + image.ConversionFormat != info.Images[(int)elements[1].Resource].ConversionFormat || + image.ShaderSwizzle != info.Images[(int)elements[1].Resource].ShaderSwizzle || + image.EmulatedCompareFunction != info.Images[(int)elements[1].Resource].EmulatedCompareFunction) + return false; + } + + var isFirst = _module.AddInstruction(SpirvOp.IEqual, _boolType, selector, UInt(0)); + var following = IAdd(UInt(elements[1].Element), _module.AddInstruction(SpirvOp.ISub, _uintType, selector, UInt(1))); + dynamicElement = _module.AddInstruction(SpirvOp.Select, _uintType, isFirst, UInt(elements[0].Element), following); + representative = elements[1]; + return true; + } + private bool TryEmitImage( Gen5ShaderInstruction instruction, Gen5ImageControl image, @@ -4905,8 +5277,30 @@ public static partial class Gen5SpirvTranslator uint imageObject; uint dstSelect; uint mipLevel; + if (_request.Memory.TryGetIndex(instruction.Pc, 0, out var runtimeMemoryIndex) && + _request.Memory[runtimeMemoryIndex] is { RuntimeDescriptor: true } runtimeEntry) { - if (TryGetImageElementCases(instruction, image, out var selector, out var elements, out error)) + return TryEmitRuntimeDescriptorImage(instruction, image, runtimeEntry, out error); + } + + { + if (TryGetImageElementCases(instruction, image, out var selector, out var elements, out error) && + TryGetDynamicImageElement(selector, elements, out var dynamicElement, out var dynamicCase)) + { + // One descriptor array read by index: the SPIR-V does not grow with the table. + if (!TryResolveLayoutImage(instruction, image, out resource, out imageObject, out dstSelect, out error, + dynamicCase, dynamicElement)) + { + return false; + } + + var dynamicMipLevel = instruction.Opcode == "ImageLoadMip" && !resource.IsStorage + ? LoadImageIntegerAddress(image, (int)ImageCoordinateComponentCount(resource)) + : UInt(0); + return EmitImageOperation(instruction, image, resource, imageObject, dstSelect, dynamicMipLevel, out error); + } + + if (error.Length == 0 && TryGetImageElementCases(instruction, image, out selector, out elements, out error)) { // One case per descriptor over a constant element, like a switch on the selector. var emitted = true; @@ -4958,6 +5352,25 @@ public static partial class Gen5SpirvTranslator } // One image operation over a resolved image object; its results are register writes. + // Sample 0 of the texel a normalized 2D coordinate falls in. + private uint FetchMultisampleTexelZero(in SpirvImageResource resource, uint image, uint coordinates) + { + var intVec2 = _module.TypeVector(_intType, 2); + var size = _module.AddInstruction(SpirvOp.ImageQuerySize, intVec2, image); + var texel = new uint[2]; + for (uint component = 0; component < 2; component++) + { + var extent = _module.AddInstruction(SpirvOp.ConvertSToF, _floatType, + _module.AddInstruction(SpirvOp.CompositeExtract, _intType, size, component)); + var position = _module.AddInstruction(SpirvOp.FMul, _floatType, + _module.AddInstruction(SpirvOp.CompositeExtract, _floatType, coordinates, component), extent); + texel[component] = _module.AddInstruction(SpirvOp.ConvertFToS, _intType, position); + } + + return _module.AddInstruction(SpirvOp.ImageFetch, resource.VectorType, image, + _module.AddInstruction(SpirvOp.CompositeConstruct, intVec2, texel), 0x40u, _module.Constant(_intType, 0)); + } + private bool EmitImageOperation( Gen5ShaderInstruction instruction, Gen5ImageControl image, @@ -4968,6 +5381,13 @@ public static partial class Gen5SpirvTranslator out string error) { error = string.Empty; + if (resource.Multisampled && + (instruction.Opcode.StartsWith("ImageGather", StringComparison.Ordinal) || + (instruction.Opcode.StartsWith("ImageSample", StringComparison.Ordinal) && resource.Arrayed))) + { + error = $"{instruction.Opcode} requires multisample sampling semantics that are not supported"; + return false; + } if (instruction.Opcode == "ImageGetResinfo") { var sizeComponentCount = ImageCoordinateComponentCount(resource); @@ -5005,6 +5425,12 @@ public static partial class Gen5SpirvTranslator _intType, size, component); + // The game reasons about its own resolution; report the guest size. + if (component < Math.Min(2u, ImageSpatialComponentCount(resource))) + { + signedValue = ScaleImageQuerySize(signedValue, resource); + } + value = Bitcast(_uintType, signedValue); } else if (component == 3) @@ -5066,6 +5492,7 @@ public static partial class Gen5SpirvTranslator var coordinateComponentCount = ImageCoordinateComponentCount(resource); var coordinates = BuildIntegerCoordinates( + resource, image, 0, coordinateComponentCount); @@ -5135,6 +5562,7 @@ public static partial class Gen5SpirvTranslator var isMax = instruction.Opcode == "ImageAtomicFmax"; var floatCoordinateCount = ImageCoordinateComponentCount(resource); var floatCoordinates = BuildIntegerCoordinates( + resource, image, 0, floatCoordinateCount); @@ -5179,6 +5607,7 @@ public static partial class Gen5SpirvTranslator var coordinateComponentCount = ImageCoordinateComponentCount(resource); var coordinates = BuildIntegerCoordinates( + resource, image, 0, coordinateComponentCount); @@ -5228,6 +5657,7 @@ public static partial class Gen5SpirvTranslator var coordinateComponentCount = ImageCoordinateComponentCount(resource); var coordinates = BuildIntegerCoordinates( + resource, image, 0, coordinateComponentCount); @@ -5244,6 +5674,7 @@ public static partial class Gen5SpirvTranslator var coordinateComponentCount = ImageCoordinateComponentCount(resource); var coordinates = BuildIntegerCoordinates( + resource, image, 0, coordinateComponentCount); @@ -5364,110 +5795,119 @@ public static partial class Gen5SpirvTranslator addressCursor, coordinateComponentCount, resource); - // Non-pixel samples require explicit derivatives or a level of detail. - // Use level zero when the instruction supplies neither. - var explicitLod = hasGradients || hasZeroLod || hasLod || - _stage != Gen5SpirvStage.Pixel; - var lod = hasZeroLod - ? Float(0) - : hasLod - ? LoadImageFloatAddress( - image, - addressCursor + (int)coordinateComponentCount) - : explicitLod - ? Float(0) - : lodOrBias; - if (hasOffset) + if (resource.Multisampled) { - // Vulkan before maintenance8 forbids the dynamic Offset - // image operand on non-gather sampling operations. RDNA - // offsets are per-lane VGPR values, so ConstOffset is not - // equivalent. Fold the texel offset into normalized sample - // coordinates using the queried mip extent instead. - var offsetLod = explicitLod && !hasGradients - ? lod - : Float(0); - coordinates = ApplyDynamicSampleOffset( - resource, - imageObject, - coordinates, - offset, - offsetLod); - } - - var imageOperands = - hasGradients ? 4u : explicitLod ? 2u : hasBias ? 1u : 0u; - var operands = new List - { - imageObject, - coordinates, - }; - - if (imageOperands != 0) - { - operands.Add(imageOperands); - if (hasGradients) - { - operands.Add(gradientX); - operands.Add(gradientY); - } - else if (explicitLod) - { - operands.Add(lod); - } - else if (hasBias) - { - operands.Add(lodOrBias); - } - - } - - if (hasCompare && resource.EmulatedCompareFunction >= 0) - { - // A color format cannot back a Vulkan depth-compare view; compare - // the sampled first channel like RDNA does for such formats. - var texel = _module.AddInstruction( - explicitLod ? SpirvOp.ImageSampleExplicitLod : SpirvOp.ImageSampleImplicitLod, - resource.VectorType, - [.. operands]); - var depth = EmulatedDepthCompare( - reference, - _module.AddInstruction(SpirvOp.CompositeExtract, _floatType, texel, 0u), - resource.EmulatedCompareFunction); - sampled = _module.AddInstruction( - SpirvOp.CompositeConstruct, - resource.VectorType, - depth, - depth, - depth, - Float(1f)); - } - else if (hasCompare) - { - // The sampler carries the compare; the depth result fills x, y, z. - var drefOperands = new List { imageObject, coordinates, reference }; - drefOperands.AddRange(operands.Skip(2)); - var depth = _module.AddInstruction( - explicitLod ? SpirvOp.ImageSampleDrefExplicitLod : SpirvOp.ImageSampleDrefImplicitLod, - _floatType, - [.. drefOperands]); - sampled = _module.AddInstruction( - SpirvOp.CompositeConstruct, - resource.VectorType, - depth, - depth, - depth, - Float(1f)); + // Vulkan cannot sample a multisample image. A sample instruction reads it as + // sample 0 of the texel the normalized coordinate falls in, as shadPS4 lowers it. + sampled = UnpackImageTexel(resource, FetchMultisampleTexelZero(resource, imageObject, coordinates)); } else { - sampled = _module.AddInstruction( - explicitLod - ? SpirvOp.ImageSampleExplicitLod - : SpirvOp.ImageSampleImplicitLod, - resource.VectorType, - [.. operands]); - sampled = UnpackImageTexel(resource, sampled); + // Non-pixel samples require explicit derivatives or a level of detail. + // Use level zero when the instruction supplies neither. + var explicitLod = hasGradients || hasZeroLod || hasLod || + _stage != Gen5SpirvStage.Pixel; + var lod = hasZeroLod + ? Float(0) + : hasLod + ? LoadImageFloatAddress( + image, + addressCursor + (int)coordinateComponentCount) + : explicitLod + ? Float(0) + : lodOrBias; + if (hasOffset) + { + // Vulkan before maintenance8 forbids the dynamic Offset + // image operand on non-gather sampling operations. RDNA + // offsets are per-lane VGPR values, so ConstOffset is not + // equivalent. Fold the texel offset into normalized sample + // coordinates using the queried mip extent instead. + var offsetLod = explicitLod && !hasGradients + ? lod + : Float(0); + coordinates = ApplyDynamicSampleOffset( + resource, + imageObject, + coordinates, + offset, + offsetLod); + } + + var imageOperands = + hasGradients ? 4u : explicitLod ? 2u : hasBias ? 1u : 0u; + var operands = new List + { + imageObject, + coordinates, + }; + + if (imageOperands != 0) + { + operands.Add(imageOperands); + if (hasGradients) + { + operands.Add(gradientX); + operands.Add(gradientY); + } + else if (explicitLod) + { + operands.Add(lod); + } + else if (hasBias) + { + operands.Add(lodOrBias); + } + + } + + if (hasCompare && resource.EmulatedCompareFunction >= 0) + { + // A color format cannot back a Vulkan depth-compare view; compare + // the sampled first channel like RDNA does for such formats. + var texel = _module.AddInstruction( + explicitLod ? SpirvOp.ImageSampleExplicitLod : SpirvOp.ImageSampleImplicitLod, + resource.VectorType, + [.. operands]); + var depth = EmulatedDepthCompare( + reference, + _module.AddInstruction(SpirvOp.CompositeExtract, _floatType, texel, 0u), + resource.EmulatedCompareFunction); + sampled = _module.AddInstruction( + SpirvOp.CompositeConstruct, + resource.VectorType, + depth, + depth, + depth, + Float(1f)); + } + else if (hasCompare) + { + // The sampler carries the compare; the depth result fills x, y, z. + var drefOperands = new List { imageObject, coordinates, reference }; + drefOperands.AddRange(operands.Skip(2)); + var depth = _module.AddInstruction( + explicitLod ? SpirvOp.ImageSampleDrefExplicitLod : SpirvOp.ImageSampleDrefImplicitLod, + _floatType, + [.. drefOperands]); + sampled = _module.AddInstruction( + SpirvOp.CompositeConstruct, + resource.VectorType, + depth, + depth, + depth, + Float(1f)); + } + else + { + sampled = _module.AddInstruction( + explicitLod + ? SpirvOp.ImageSampleExplicitLod + : SpirvOp.ImageSampleImplicitLod, + resource.VectorType, + [.. operands]); + sampled = UnpackImageTexel(resource, sampled); + } } } else if (instruction.Opcode.StartsWith( @@ -6140,16 +6580,22 @@ public static partial class Gen5SpirvTranslator } private uint BuildIntegerCoordinates( + in SpirvImageResource resource, Gen5ImageControl image, int start, uint componentCount) { + // Only the two scaled axes move; an array layer or a volume slice is not a pixel. + var scaledComponents = start == 0 ? Math.Min(2, (int)ImageSpatialComponentCount(resource)) : 0; var components = new uint[checked((int)componentCount)]; for (var component = 0; component < components.Length; component++) { - components[component] = Bitcast( + var value = Bitcast( _intType, LoadImageIntegerAddress(image, start + component)); + components[component] = component < scaledComponents + ? ScaleIntegerImageCoordinate(value, resource) + : value; } if (componentCount == 1) @@ -6279,6 +6725,30 @@ public static partial class Gen5SpirvTranslator SpirvOp.ConvertSToF, spatialFloatType, texelOffset); + // The instruction's offset is in guest texels. Convert it to host texels + // before normalizing by the host mip extent. Array layers and depth are + // never scaled; the per-image mask also preserves unscaled bindings. + if (UsesRenderScale) + { + var factor = LoadImageScaleFactor(resource, ImageCoordinateFactorDword); + if (spatialComponentCount == 1) + { + offsetFloat = _module.AddInstruction(SpirvOp.FMul, _floatType, offsetFloat, factor); + } + else + { + var components = new uint[checked((int)spatialComponentCount)]; + for (uint component = 0; component < spatialComponentCount; component++) + { + var value = _module.AddInstruction(SpirvOp.CompositeExtract, _floatType, offsetFloat, component); + components[component] = component < 2 + ? _module.AddInstruction(SpirvOp.FMul, _floatType, value, factor) + : value; + } + offsetFloat = _module.AddInstruction(SpirvOp.CompositeConstruct, spatialFloatType, components); + } + } + var normalizedOffset = _module.AddInstruction( SpirvOp.FDiv, spatialFloatType, @@ -6320,6 +6790,10 @@ public static partial class Gen5SpirvTranslator arrayOffset); } + private uint PixelSampleMaskOutputElement() => _module.AddInstruction( + SpirvOp.AccessChain, _module.TypePointer(SpirvStorageClass.Output, _intType), + _pixelSampleMaskOutput, UInt(0)); + private bool TryEmitExport( Gen5ShaderInstruction instruction, Gen5ExportControl export, @@ -6342,6 +6816,40 @@ public static partial class Gen5SpirvTranslator Store(_pixelValidMaskActive, Load(_boolType, _exec)); } + if (export.Target == 8) + { + if (_pixelDepthOutput != 0 && (export.EnableMask & 1) != 0) + { + if (export.Compressed) + { + error = "compressed MRTZ cannot export floating-point depth"; + return false; + } + var depth = Bitcast(_floatType, LoadV(instruction.Sources[0].Value)); + Store(_pixelDepthOutput, _module.AddInstruction(SpirvOp.Select, _floatType, + Load(_boolType, _exec), depth, Load(_floatType, _pixelDepthOutput))); + } + if (_pixelSampleMaskOutput != 0 && (export.EnableMask & 4) != 0) + { + // Compressed MRTZ packs the sample mask into Y[15:0]; it is + // an integer mask, not a pair of half-precision colors. + var mask = LoadV(instruction.Sources[export.Compressed ? 1 : 2].Value); + if (export.Compressed) + mask = BitwiseAnd(mask, UInt(0xFFFF)); + if (_request.PixelMaskExportSamples == 1 && _request.PixelRasterizationSamples == 2) + { + mask = _module.AddInstruction(SpirvOp.Select, _uintType, + _module.AddInstruction(SpirvOp.INotEqual, _boolType, + BitwiseAnd(mask, UInt(1)), UInt(0)), UInt(3), UInt(0)); + } + var pointer = PixelSampleMaskOutputElement(); + var value = _module.AddInstruction(SpirvOp.Select, _intType, + Load(_boolType, _exec), Bitcast(_intType, mask), Load(_intType, pointer)); + Store(pointer, value); + } + return true; + } + if (!_pixelOutputs.TryGetValue(export.Target, out var output)) { return true; @@ -6383,21 +6891,7 @@ public static partial class Gen5SpirvTranslator if (export.Compressed) { - var value = LoadCompressedExportComponent( - instruction, - component); - values[component] = output.Kind switch - { - Gen5PixelOutputKind.Uint => _module.AddInstruction( - SpirvOp.ConvertFToU, - _uintType, - value), - Gen5PixelOutputKind.Sint => _module.AddInstruction( - SpirvOp.ConvertFToS, - _intType, - value), - _ => value, - }; + values[component] = LoadPixelCompressedExportComponent(instruction, component, output); continue; } @@ -6535,7 +7029,7 @@ public static partial class Gen5SpirvTranslator return true; } - if (_stage != Gen5SpirvStage.Vertex) + if (!HasVertexOutputs) { return true; } @@ -7232,6 +7726,45 @@ public static partial class Gen5SpirvTranslator _request.Program.Address == address; } + private uint LoadPixelCompressedExportComponent( + Gen5ShaderInstruction instruction, int component, SpirvPixelOutput output) + { + if (output.ExportFormat is Gen5PixelExportFormat.Uint16 or Gen5PixelExportFormat.Sint16) + { + var signed = output.ExportFormat == Gen5PixelExportFormat.Sint16; + var packed = LoadV(instruction.Sources[component >> 1].Value); + var value = _module.AddInstruction( + signed ? SpirvOp.BitFieldSExtract : SpirvOp.BitFieldUExtract, + signed ? _intType : _uintType, + signed ? Bitcast(_intType, packed) : packed, + UInt((uint)(component & 1) * 16), UInt(16)); + return output.Kind switch + { + Gen5PixelOutputKind.Uint => signed ? Bitcast(_uintType, value) : value, + Gen5PixelOutputKind.Sint => signed ? value : Bitcast(_intType, value), + _ => _module.AddInstruction(signed ? SpirvOp.ConvertSToF : SpirvOp.ConvertUToF, _floatType, value), + }; + } + + uint decoded; + if (output.ExportFormat is Gen5PixelExportFormat.Unorm16 or Gen5PixelExportFormat.Snorm16) + { + var unpacked = Ext(output.ExportFormat == Gen5PixelExportFormat.Unorm16 ? 61u : 60u, + _vec2Type, LoadV(instruction.Sources[component >> 1].Value)); + decoded = _module.AddInstruction(SpirvOp.CompositeExtract, _floatType, unpacked, (uint)(component & 1)); + } + else + { + decoded = LoadCompressedExportComponent(instruction, component); + } + return output.Kind switch + { + Gen5PixelOutputKind.Uint => _module.AddInstruction(SpirvOp.ConvertFToU, _uintType, decoded), + Gen5PixelOutputKind.Sint => _module.AddInstruction(SpirvOp.ConvertFToS, _intType, decoded), + _ => decoded, + }; + } + private uint LoadCompressedExportComponent( Gen5ShaderInstruction instruction, int component) @@ -7410,6 +7943,14 @@ public static partial class Gen5SpirvTranslator private uint LoadBufferWord(int binding, uint dwordAddress) { + // With the device measured to return zero past the end of a descriptor range, the + // range test, the address clamp and the zero select only reproduce what the read + // already does. The guest expects zero there, and so the load stands alone. + if (_zeroOutOfBoundsBufferReads) + { + return Load(_uintType, BufferWordPointer(binding, dwordAddress)); + } + var inRange = IsBufferWordInRange(binding, dwordAddress); var safeAddress = _module.AddInstruction( SpirvOp.Select, @@ -7426,6 +7967,20 @@ public static partial class Gen5SpirvTranslator UInt(0)); } + // A buffer's element stride: from shader data when the host writes it there + // (BindingLayout.UsesRuntimeBufferStrides), else compiled from the specialization. + private uint BufferStride(int binding, uint packedStride) + { + var layout = _request.Bindings; + if (!layout.UsesRuntimeBufferStrides || (packedStride & BufferSpecialization.SwizzleEnabledBit) != 0) + { + return UInt(packedStride & BufferSpecialization.StrideMask); + } + + var packed = LoadShaderDataDword(UInt(layout.BufferStrideDword + (uint)binding / 2)); + return BitwiseAnd(ShiftRightLogical(packed, UInt(((uint)binding % 2) * 16)), UInt(BufferSpecialization.StrideMask)); + } + private uint ApplyGuestBufferByteBias(int binding, uint byteAddress) { { @@ -7441,23 +7996,35 @@ public static partial class Gen5SpirvTranslator () => Store(BufferWordPointer(binding, dwordAddress), value)); } - private uint IsBufferWordInRange(int binding, uint dwordAddress) + private bool BufferMayHaveTailPadding(int binding) => + _request.Bindings.UsesRuntimeBufferStrides && + (_request.Resources.Info.Buffers[binding].Formatted || !_request.Resources.Info.Buffers[binding].DwordAddressed); + + private uint BufferByteLength(int binding) + { + var packed = LoadShaderDataDword(UInt(_request.Bindings.BufferStrideDword + (uint)binding / 2)); + var padding = BitwiseAnd(ShiftRightLogical(packed, UInt(((uint)binding % 2) * 16 + 14)), UInt(3)); + return _module.AddInstruction(SpirvOp.ISub, _uintType, ShiftLeftLogical(BufferWordCount(binding), UInt(2)), padding); + } + + // The hardware range-checks an access by its first byte: a word that starts inside the + // guest range is read and written whole, even when the range ends inside it. Runtime + // views are padded to whole words, so their word count is exactly that bound. + private uint IsBufferWordInRange(int binding, uint dwordAddress) => + _module.AddInstruction(SpirvOp.ULessThan, _boolType, dwordAddress, BufferWordCount(binding)); + + private uint BufferWordCount(int binding) { var buffer = _module.AddInstruction( SpirvOp.AccessChain, _storageBlockPointer, _globalBuffers, UInt((uint)binding)); - var length = _module.AddInstruction( + return _module.AddInstruction( SpirvOp.ArrayLength, _uintType, buffer, 0); - return _module.AddInstruction( - SpirvOp.ULessThan, - _boolType, - dwordAddress, - length); } private uint BufferWordPointer(int binding, uint dwordAddress) => @@ -7469,6 +8036,74 @@ public static partial class Gen5SpirvTranslator UInt(0), dwordAddress); + private uint _functionUintPointer; + private uint _functionVec2Pointer; + private uint _functionBoolPointer; + + // Register files and state live in the function executing the program. + // The title-state diagnostic alone needs them shared with the entry-point wrapper. + private bool _functionScopeState; + + // Declare variables in the executing function's first block, or as Private globals + // when the title-state diagnostic needs to inspect them from the wrapper. + private void DeclareRegisterFiles() + { + uint Variable(uint valueType, uint initializer) + { + if (_functionScopeState) + { + return _module.AddFunctionVariable(_module.TypePointer(SpirvStorageClass.Function, valueType), initializer); + } + + var variable = _module.AddGlobalVariable( + _module.TypePointer(SpirvStorageClass.Private, valueType), + SpirvStorageClass.Private, + initializer); + _interfaces.Add(variable); + return variable; + } + + uint Flag(bool initialValue) + { + var variable = Variable(_uintType, UInt(initialValue ? 1u : 0u)); + _flagVariables.Add(variable); + return variable; + } + + // An unbounded relative write can address a register that no static + // operand names. Preserve the complete bank for those programs. + if (_needsDynamicVectorArray) + { + var arrayType = _module.TypeArray(_uintType, VectorRegisterCount); + _vectorRegisters = Variable(arrayType, _module.ConstantNull(arrayType)); + _module.AddName(_vectorRegisters, "vgpr"); + } + + _scc = Flag(false); + _vcc = Flag(false); + _reachedPixelExport = Flag(false); + + _programCounter = Variable(_uintType, _module.Constant(_uintType, 0)); + _programActive = Flag(true); + if (UsesWave64Exchange()) + { + _wave64ExchangeParity = Variable(_uintType, _module.Constant(_uintType, 0)); + _module.AddName(_wave64ExchangeParity, "wave64ExchangeParity"); + } + if (_maxDispatcherSteps > 0) + { + _iterationGuard = Variable(_uintType, _module.Constant(_uintType, 0)); + _module.AddName(_iterationGuard, "pcGuard"); + } + + + foreach (var slot in FindLaneSpillSlots()) + { + var variable = Variable(_uintType, _module.Constant(_uintType, 0)); + _module.AddName(variable, $"v{slot.Register}_lane{slot.Lane}"); + _laneSpillSlots.Add(slot, variable); + } + } // Without subgroup operations each invocation is a one-lane wave, so a V_WRITELANE to // lane N > 0 used to be dropped and V_READLANE returned lane 0. Compilers spill SGPRs // (EXEC included) to VGPR lanes like this; restoring the wrong EXEC made waterfall @@ -7478,7 +8113,7 @@ public static partial class Gen5SpirvTranslator var slots = new List<(uint Register, uint Lane)>(); if (_emulateWave64) { - return slots; + return FindPureLaneSpillSlots(); } var ownsLaneZero = !UsesSubgroupOperations(); @@ -7504,6 +8139,70 @@ public static partial class Gen5SpirvTranslator return slots; } + // Under wave64 emulation a lane read normally rendezvous with the other host subgroup through + // LDS and a workgroup barrier. A VGPR written only by constant-lane V_WRITELANE holds spilled + // scalars, which are wave-uniform, so every invocation can keep each written lane itself and a + // constant-lane V_READLANE needs no exchange. V_WRITELANE still updates the VGPR, so other + // reads of it are unaffected. Relative VGPR addressing could alias any register, so programs + // that use it keep the exchange. + private List<(uint Register, uint Lane)> FindPureLaneSpillSlots() + { + var slots = new List<(uint Register, uint Lane)>(); + var instructions = _request.Program.Instructions; + if (instructions.Any(static instruction => instruction.Opcode.StartsWith("VMovrel", StringComparison.Ordinal))) + { + return slots; + } + + var candidates = new HashSet(); + var rejected = new HashSet(); + foreach (var instruction in instructions) + { + var isWritelane = instruction.Opcode == "VWritelaneB32"; + var isReadlane = instruction.Opcode == "VReadlaneB32"; + for (var index = 0; index < instruction.Destinations.Count; index++) + { + var operand = instruction.Destinations[index]; + if (operand.Kind != Gen5OperandKind.VectorRegister) + { + continue; + } + + if (isWritelane && index == 0 && TryGetConstantLane(instruction, out _)) + { + candidates.Add(operand.Value); + } + else + { + rejected.Add(operand.Value); + } + } + + // A variable lane read cannot use a slot; it keeps reading the VGPR, which V_WRITELANE + // still writes, so ordinary vector reads of the register stay exact as well. + if (isReadlane && instruction.Sources.Count > 0 && + instruction.Sources[0].Kind == Gen5OperandKind.VectorRegister && + !TryGetConstantLane(instruction, out _)) + { + rejected.Add(instruction.Sources[0].Value); + } + } + + foreach (var instruction in instructions) + { + if (instruction.Opcode == "VWritelaneB32" && + TryGetVectorDestination(instruction, out var register) && + candidates.Contains(register) && !rejected.Contains(register) && + TryGetConstantLane(instruction, out var lane) && + !slots.Contains((register, lane))) + { + slots.Add((register, lane)); + } + } + + return slots; + } + // The lane select operand (source 1) of V_READLANE/V_WRITELANE when it is a constant. private bool TryGetConstantLane(Gen5ShaderInstruction instruction, out uint lane) { @@ -7578,8 +8277,10 @@ public static partial class Gen5SpirvTranslator { if (!variables.TryGetValue(register, out var variable)) { - variable = _module.AddGlobalVariable(_privateUintPointer, SpirvStorageClass.Private, _module.ConstantNull(_uintType)); - _interfaces.Add(variable); + variable = _functionScopeState + ? _module.AddFunctionVariable(_functionUintPointer, UInt(0)) + : _module.AddGlobalVariable(_privateUintPointer, SpirvStorageClass.Private, UInt(0)); + if (!_functionScopeState) _interfaces.Add(variable); _module.AddName(variable, $"{prefix}{register}"); variables.Add(register, variable); } @@ -7597,13 +8298,7 @@ public static partial class Gen5SpirvTranslator private uint VectorPointer(uint register) { if (_vectorRegisters != 0) - { - return _module.AddInstruction( - SpirvOp.AccessChain, - _privateUintPointer, - _vectorRegisters, - UInt(register)); - } + return _module.AddInstruction(SpirvOp.AccessChain, _functionUintPointer, _vectorRegisters, UInt(register)); if (register >= VectorRegisterCount) { @@ -7618,89 +8313,43 @@ public static partial class Gen5SpirvTranslator // takes a computed index instead of a constant. The index is masked to // the array bounds: SPIR-V leaves an out-of-range Private access chain // undefined, and a mask costs nothing next to the surrounding load. - // Every register a relative access can reach: from the lowest V_MOVREL* base to past the - // highest register the program names (a multi-dword result may extend beyond its listed - // first register). An index outside it reads zero and writes nothing. - private static (uint First, uint Last) DynamicVectorRange(Gen5ShaderProgram program) - { - var first = VectorRegisterCount - 1; - var last = 0u; - foreach (var instruction in program.Instructions) - { - foreach (var operand in instruction.Sources.Concat(instruction.Destinations)) - { - if (operand.Kind == Gen5OperandKind.VectorRegister) - { - last = Math.Max(last, operand.Value); - if (instruction.Opcode.StartsWith("VMovrel", StringComparison.Ordinal)) - { - first = Math.Min(first, operand.Value); - } - } - } - } + private uint DynamicVectorPointer(uint registerIndex) => + _module.AddInstruction(SpirvOp.AccessChain, _functionUintPointer, + _vectorRegisters, BitwiseAnd(registerIndex, UInt(VectorRegisterCount - 1))); - last = Math.Min(last + 8, VectorRegisterCount - 1); - return (Math.Min(first, last), last); - } - - private uint LoadVDynamic(uint registerIndex) - { - var result = UInt(0); - for (var register = _dynamicVectorRange.First; register <= _dynamicVectorRange.Last; register++) - { - result = _module.AddInstruction( - SpirvOp.Select, - _uintType, - _module.AddInstruction(SpirvOp.IEqual, _boolType, registerIndex, UInt(register)), - Load(_uintType, VectorPointer(register)), - result); - } - - return result; - } + private uint LoadVDynamic(uint registerIndex) => + Load(_uintType, DynamicVectorPointer(registerIndex)); private void StoreVDynamic(uint registerIndex, uint value) { - // With EXEC known to be all ones the write needs no EXEC test. - var exec = _execKnownFull ? 0 : Load(_boolType, _exec); - for (var register = _dynamicVectorRange.First; register <= _dynamicVectorRange.Last; register++) - { - var pointer = VectorPointer(register); - var selected = _module.AddInstruction(SpirvOp.IEqual, _boolType, registerIndex, UInt(register)); - if (exec != 0) - { - selected = _module.AddInstruction(SpirvOp.LogicalAnd, _boolType, exec, selected); - } - Store(pointer, _module.AddInstruction(SpirvOp.Select, _uintType, selected, value, Load(_uintType, pointer))); - } + var pointer = DynamicVectorPointer(registerIndex); + if (!_execKnownFull) + value = _module.AddInstruction(SpirvOp.Select, _uintType, + Load(_boolType, _exec), value, Load(_uintType, pointer)); + if (_executingWave != 0) value = GuardCooperativeUint(value, Load(_uintType, pointer)); + Store(pointer, value); } - private uint PackedHalfPointer(uint register) => - _module.AddInstruction( - SpirvOp.AccessChain, - _privateVec2Pointer, - PackedHalfRegisters(), - UInt(register)); - - // Declared on first use when Private. AMD's compiler keeps an unused 4 KiB private - // array as a named .bss global: two stages then fail to link, and the driver copies - // the NOBITS section as file data and reads past the end of the ELF. - private uint PackedHalfRegisters() + // One variable per register, declared on first use. The packed-half shadow of the vector + // register file is only ever addressed by a constant register number, and a 512-entry + // array is the shape that makes Metal keep the file in thread memory instead of + // registers - the same trap V_MOVREL hit. A Private array also stayed as a named .bss + // global on AMD, where two stages then failed to link because the driver copied the + // NOBITS section as file data and read past the end of the ELF. + private uint PackedHalfPointer(uint register) { - if (_packedHalfRegisters != 0) + if (_packedHalfRegisterVariables.TryGetValue(register, out var variable)) { - return _packedHalfRegisters; + return variable; } - var arrayType = _module.TypeArray(_vec2Type, VectorRegisterCount); - _packedHalfRegisters = _module.AddGlobalVariable( - _module.TypePointer(SpirvStorageClass.Private, arrayType), - SpirvStorageClass.Private, - _module.ConstantNull(arrayType)); - _interfaces.Add(_packedHalfRegisters); - _module.AddName(_packedHalfRegisters, "vgprPackedHalf"); - return _packedHalfRegisters; + variable = _functionScopeState + ? _module.AddFunctionVariable(_functionVec2Pointer, _module.ConstantNull(_vec2Type)) + : _module.AddGlobalVariable(_privateVec2Pointer, SpirvStorageClass.Private, _module.ConstantNull(_vec2Type)); + _packedHalfRegisterVariables.Add(register, variable); + if (!_functionScopeState) _interfaces.Add(variable); + _module.AddName(variable, $"vgprPackedHalf{register}"); + return variable; } private uint LoadS(uint register) => Load(_uintType, ScalarPointer(register)); @@ -7709,7 +8358,9 @@ public static partial class Gen5SpirvTranslator private void StoreS(uint register, uint value) { + if (_executingWave != 0) value = GuardCooperativeUint(value, LoadS(register)); Store(ScalarPointer(register), value); + if (register is 106 or 107) { if (_waveLaneCount != 32 || register == 106) @@ -7732,6 +8383,25 @@ public static partial class Gen5SpirvTranslator { if (_waveLaneCount != 32) { + // The lane's bit lives in one of the two mask registers, so select that register + // and shift inside it. Composing the 64-bit mask and masking it with a 64-bit + // lane bit costs several times as much on a host that emulates 64-bit integers, + // and VCC and EXEC refresh their lane flag on every mask write. + if (_subgroupInvocationIdInput != 0 && _emulateWave64) + { + var lane = GuestWaveLane(); + var word = _module.AddInstruction( + SpirvOp.Select, + _uintType, + _module.AddInstruction(SpirvOp.ULessThan, _boolType, lane, UInt(32)), + LoadS(lowRegister), + LoadS(lowRegister + 1)); + return IsNotZero( + BitwiseAnd( + ShiftRightLogical(word, BitwiseAnd(lane, UInt(31))), + UInt(1))); + } + return IsWaveMaskActive(LoadS64(lowRegister)); } @@ -7776,11 +8446,15 @@ public static partial class Gen5SpirvTranslator oldValue); } + if (_executingWave != 0) value = GuardCooperativeUint(value, LoadV(register)); Store(VectorPointer(register), value); + } private void StorePackedHalf(uint register, uint value) { + if (_executingWave != 0) value = _module.AddInstruction(SpirvOp.Select, _vec2Type, + _executingWave, value, Load(_vec2Type, PackedHalfPointer(register))); if (_execKnownFull) { Store(PackedHalfPointer(register), value); @@ -7843,6 +8517,11 @@ public static partial class Gen5SpirvTranslator private void Store(uint pointer, uint value) { + if (_executingWave != 0 && pointer == _exec) value = LogicalAnd(value, _executingWave); + if (_executingWave != 0 && (pointer == _scc || pointer == _programActive)) + value = _module.AddInstruction(SpirvOp.Select, _boolType, _executingWave, value, Load(_boolType, pointer)); + if (_executingWave != 0 && pointer == _programCounter) + value = GuardCooperativeUint(value, Load(_uintType, pointer)); if (_flagVariables.Contains(pointer)) { value = _module.AddInstruction( @@ -8017,6 +8696,47 @@ public static partial class Gen5SpirvTranslator ? value : _module.AddInstruction(SpirvOp.GroupNonUniformShuffle, _uintType, UInt(3), value, lane); + // Fragment subgroups are allowed to be only partly populated. The missing + // lanes still exist on the guest wave, while SPIR-V cannot shuffle from + // them. For a graphics guest wave, make that read a duplicate of the + // caller's value. This is neutral for the idempotent lane reductions used + // by the guest (min/max/and/or), unlike the undefined SPIR-V result or the + // old zero substitute. A physically present lane remains readable even if + // guest EXEC disabled it; callers decide whether that needs a zero result. + private uint ShuffleGuestLaneOrSelf(uint value, uint guestLane, out uint sourcePresent) + { + if (_stage == Gen5SpirvStage.Compute || _subgroupInvocationIdInput == 0) + { + sourcePresent = _module.ConstantBool(true); + return ShuffleHalfWaveLane(value, guestLane); + } + + var sourceLane = BitwiseAnd(guestLane, UInt(31)); + var presenceMask = OwnHalfBallot(_module.ConstantBool(true)); + sourcePresent = IsNotZero(BitwiseAnd( + presenceMask, + ShiftLeftLogical(UInt(1), sourceLane))); + var currentLane = Load(_uintType, _subgroupInvocationIdInput); + var groupBase = BitwiseAnd(currentLane, UInt(~31u)); + var sourceHostLane = IAdd(groupBase, sourceLane); + var safeSourceLane = _module.AddInstruction( + SpirvOp.Select, + _uintType, + sourcePresent, + sourceHostLane, + currentLane); + return ShuffleLane(value, safeSourceLane); + } + + // DPP row/quad operations and PERMLANE16 never cross a 32-lane half-wave. + // Guest lanes 32..63 belong to the second host subgroup on 32-lane devices; + // using those guest indices directly in OpGroupNonUniformShuffle is undefined. + // Preserve the current physical half on devices whose subgroup is wider than 32. + private uint ShuffleHalfWaveLane(uint value, uint guestLane) => + _subgroupInvocationIdInput == 0 ? value : ShuffleLane(value, + BitwiseOr(BitwiseAnd(Load(_uintType, _subgroupInvocationIdInput), UInt(0xFFFF_FFE0)), + BitwiseAnd(guestLane, UInt(31)))); + private uint CurrentLaneBit() { if (_subgroupInvocationIdInput == 0) @@ -8150,9 +8870,12 @@ public static partial class Gen5SpirvTranslator { var parity = Load(_uintType, _wave64ExchangeParity); Store(_wave64ExchangeParity, BitwiseXor(parity, UInt(1))); - return IAdd( + var exchange = IAdd( UInt(_wave64ExchangeOffset), _module.AddInstruction(SpirvOp.IMul, _uintType, parity, UInt(Wave64ExchangeSlotCount))); + return _request.CooperativeWave64Workgroup + ? IAdd(exchange, _module.AddInstruction(SpirvOp.IMul, _uintType, CooperativeWaveId(), UInt(Wave64ExchangeDwordCount))) + : exchange; } private uint Wave64ExchangePointer(uint exchange, uint slot) => @@ -8234,6 +8957,35 @@ public static partial class Gen5SpirvTranslator return _halfMaskPlan; } + // Null until a wait is seen in the current block; then true once any wait in the block + // drained vector memory (vmcnt(0), or vscnt(0), which is what covers stores on GFX10), + // which is how the guest publishes buffer and image stores before a barrier. It stays + // true for the rest of the block: a later LDS-only wait does not unpublish them. + private bool? _vectorMemoryWaited; + + private void NoteWaitCount(Gen5ShaderInstruction instruction) + { + if (instruction.Words.Count == 0) + { + return; + } + + var word = instruction.Words[0]; + if (instruction.Encoding == Gen5ShaderEncoding.Sopk) + { + // SOPK waits name one counter: 0x17 vscnt, 0x18 vmcnt, 0x19 expcnt, 0x1A lgkmcnt. + var counter = (word >> 23) & 0x1Fu; + var drainsVectorMemory = counter is 0x17 or 0x18 && (word & 0xFFFFu) == 0; + _vectorMemoryWaited = _vectorMemoryWaited == true || drainsVectorMemory; + return; + } + + // SOPP S_WAITCNT packs vmcnt across bits [3:0] and [15:14]. + var immediate = word & 0xFFFFu; + var vectorCount = (immediate & 0xFu) | (((immediate >> 14) & 0x3u) << 4); + _vectorMemoryWaited = _vectorMemoryWaited == true || vectorCount == 0; + } + private void EmitWave64Barrier() { var workgroup = UInt(2); @@ -8294,6 +9046,7 @@ public static partial class Gen5SpirvTranslator private void EmitExecConditional(Action emit) { var active = Load(_boolType, _exec); + if (_executingWave != 0) active = LogicalAnd(active, _executingWave); EmitConditional(active, emit); } @@ -8361,7 +9114,7 @@ public static partial class Gen5SpirvTranslator private bool UsesSubgroupOperations() => _enableGraphicsSubgroupOperations && - (UsesSubgroupShuffle() || + (_request.CooperativeWave64Workgroup || UsesSubgroupShuffle() || UsesSubgroupBroadcast() || UsesWaveControl() || _request.Program.Instructions.Any(static instruction => @@ -8414,7 +9167,7 @@ public static partial class Gen5SpirvTranslator return true; } - private static IReadOnlyList BuildBasicBlocks( + private IReadOnlyList BuildBasicBlocks( IReadOnlyList instructions) { if (instructions.Count == 0) @@ -8426,6 +9179,11 @@ public static partial class Gen5SpirvTranslator for (var index = 0; index < instructions.Count; index++) { var instruction = instructions[index]; + if (_request.CooperativeWave64Workgroup && instruction.Opcode == "SBarrier") + { + leaders.Add(instruction.Pc); + if (index + 1 < instructions.Count) leaders.Add(instructions[index + 1].Pc); + } if (IsBranch(instruction.Opcode) && TryGetBranchTargetPc(instruction, out var targetPc)) { diff --git a/src/SharpEmu.ShaderCompiler.Vulkan/Gen5TessellationBridge.cs b/src/SharpEmu.ShaderCompiler.Vulkan/Gen5TessellationBridge.cs new file mode 100644 index 0000000..fd42d7e --- /dev/null +++ b/src/SharpEmu.ShaderCompiler.Vulkan/Gen5TessellationBridge.cs @@ -0,0 +1,102 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.ShaderCompiler.Resources; + +namespace SharpEmu.ShaderCompiler.Vulkan; + +// The guest HS has already computed its factors and control points in a merged +// compute workgroup. This stage feeds those factors to Vulkan's tessellator; +// the domain shader still reads the original guest control-point ring. +public static class Gen5TessellationBridge +{ + public static uint FactorCount(Gen5TessellationDomain domain) => domain switch + { + Gen5TessellationDomain.Isolines => 2, + Gen5TessellationDomain.Triangles => 4, + Gen5TessellationDomain.Quads => 6, + _ => throw new ArgumentOutOfRangeException(nameof(domain)), + }; + + public static Gen5SpirvShader CompileControl(BindingLayout layout, Gen5TessellationDomain domain) + { + if (!layout.UsesTessellationData) + throw new ArgumentException("The tessellation control bridge requires runtime tessellation data.", nameof(layout)); + var factors = FactorCount(domain); + var module = new SpirvModuleBuilder(); + module.AddCapability(SpirvCapability.Shader); + module.AddCapability(SpirvCapability.Tessellation); + module.AddCapability(SpirvCapability.Int64); + module.AddCapability(SpirvCapability.PhysicalStorageBufferAddresses); + module.SetPhysicalStorageBuffer64MemoryModel(); + var uintType = module.TypeInt(32, false); + var ulongType = module.TypeInt(64, false); + var floatType = module.TypeFloat(32); + var boolType = module.TypeBool(); + uint UInt(uint value) => module.Constant(uintType, value); + uint Op(SpirvOp op, uint type, params uint[] values) => module.AddInstruction(op, type, values); + + var words = module.TypeRuntimeArray(uintType); + module.AddDecoration(words, SpirvDecoration.ArrayStride, 4); + var block = module.TypeStruct(words); + module.AddDecoration(block, SpirvDecoration.Block); + module.AddMemberDecoration(block, 0, SpirvDecoration.Offset, 0); + var data = module.AddGlobalVariable(module.TypePointer(SpirvStorageClass.StorageBuffer, block), SpirvStorageClass.StorageBuffer); + module.AddDecoration(data, SpirvDecoration.NonWritable); + module.AddDecoration(data, SpirvDecoration.DescriptorSet, layout.UsesBindlessImages ? 1u : 0u); + // Shares the domain stage's runtime data, not its guest SGPR inputs. + module.AddDecoration(data, SpirvDecoration.Binding, + BindingLayout.NativeBindingIndex(ShaderStage.TessellationEvaluation, DescriptorBindingKind.ShaderData)); + var patch = module.AddGlobalVariable(module.TypePointer(SpirvStorageClass.Input, uintType), SpirvStorageClass.Input); + module.AddDecoration(patch, SpirvDecoration.BuiltIn, (uint)SpirvBuiltIn.PrimitiveId); + var outputFloat = module.TypePointer(SpirvStorageClass.Output, floatType); + uint Output(SpirvBuiltIn builtIn, uint count) + { + var output = module.AddGlobalVariable(module.TypePointer(SpirvStorageClass.Output, module.TypeArray(floatType, count)), SpirvStorageClass.Output); + module.AddDecoration(output, SpirvDecoration.BuiltIn, (uint)builtIn); + module.AddDecoration(output, SpirvDecoration.Patch); + return output; + } + var outer = Output(SpirvBuiltIn.TessLevelOuter, 4); + var inner = Output(SpirvBuiltIn.TessLevelInner, 2); + var voidType = module.TypeVoid(); + var entry = module.BeginFunction(voidType, module.TypeFunction(voidType)); + module.AddLabel(); + uint Word(uint index) => Op(SpirvOp.Load, uintType, Op(SpirvOp.AccessChain, + module.TypePointer(SpirvStorageClass.StorageBuffer, uintType), data, UInt(0), UInt(layout.TessellationDataDword + index))); + var addressLow = Op(SpirvOp.UConvert, ulongType, Word(Gen5TessellationData.FactorAddress)); + var addressHigh = Op(SpirvOp.ShiftLeftLogical, ulongType, + Op(SpirvOp.UConvert, ulongType, Word(Gen5TessellationData.FactorAddress + 1)), UInt(32)); + var address = Op(SpirvOp.BitwiseOr, ulongType, addressLow, addressHigh); + var first = Op(SpirvOp.IMul, uintType, Op(SpirvOp.Load, uintType, patch), UInt(factors * 4)); + var minimum = Op(SpirvOp.Bitcast, floatType, Word(Gen5TessellationData.MinimumLevel)); + var maximum = Op(SpirvOp.Bitcast, floatType, Word(Gen5TessellationData.MaximumLevel)); + var zero = module.ConstantFloat(floatType, 0); + void Write(uint output, uint index, uint? sourceIndex) + { + var value = zero; + if (sourceIndex is { } source) + { + var offset = Op(SpirvOp.IAdd, uintType, first, UInt(source * 4)); + var pointer = Op(SpirvOp.ConvertUToPtr, module.TypePointer(SpirvStorageClass.PhysicalStorageBuffer, floatType), + Op(SpirvOp.IAdd, ulongType, address, Op(SpirvOp.UConvert, ulongType, offset))); + var level = Op(SpirvOp.Load, floatType, pointer, 2, 4); + var lower = Op(SpirvOp.Select, floatType, Op(SpirvOp.FOrdLessThan, boolType, level, minimum), minimum, level); + var clamped = Op(SpirvOp.Select, floatType, Op(SpirvOp.FOrdGreaterThan, boolType, lower, maximum), maximum, lower); + // Nonpositive or NaN outer factors must still cull the patch. + value = Op(SpirvOp.Select, floatType, Op(SpirvOp.FOrdGreaterThan, boolType, level, zero), clamped, level); + } + module.AddStatement(SpirvOp.Store, Op(SpirvOp.AccessChain, outputFloat, output, UInt(index)), value); + } + var outerCount = domain == Gen5TessellationDomain.Isolines ? 2u : domain == Gen5TessellationDomain.Triangles ? 3u : 4u; + for (uint index = 0; index < 4; index++) + Write(outer, index, index < outerCount ? domain == Gen5TessellationDomain.Isolines ? 1u - index : index : null); + for (uint index = 0; index < 2; index++) + Write(inner, index, index + outerCount < factors ? index + outerCount : null); + module.AddStatement(SpirvOp.Return); + module.EndFunction(); + module.AddEntryPoint(SpirvExecutionModel.TessellationControl, entry, "main", [data, patch, outer, inner]); + module.AddExecutionMode(entry, SpirvExecutionMode.OutputVertices, 1); + return new(module.Build(), 0); + } +} diff --git a/src/SharpEmu.ShaderCompiler.Vulkan/SpirvFixedShaders.cs b/src/SharpEmu.ShaderCompiler.Vulkan/SpirvFixedShaders.cs index 560daa9..062d0ad 100644 --- a/src/SharpEmu.ShaderCompiler.Vulkan/SpirvFixedShaders.cs +++ b/src/SharpEmu.ShaderCompiler.Vulkan/SpirvFixedShaders.cs @@ -420,6 +420,137 @@ public static class SpirvFixedShaders return module.Build(); } + // The f16 conversion probe: for input[i] at binding 0, write two dwords at binding 1, + // output[2i] = the f32 bits of GLSL UnpackHalf2x16(input[i] & 0xFFFF).x and + // output[2i + 1] = GLSL PackHalf2x16(vec2(asfloat(input[i]), 0)) & 0xFFFF. The host + // compares both against the integer sequences the translator emits by default, and + // only then lets the translator use these two ext instructions. 64 threads per group, + // one per value; the input is padded to a multiple of 64 so no thread is out of range. + // Copies binding 0 to binding 1 with no bounds check at all, so a read past the descriptor + // range reaches the device and the caller can see what it returns. + public static byte[] CreateUncheckedBufferCopyProbe() + { + var module = new SpirvModuleBuilder(); + module.AddCapability(SpirvCapability.Shader); + + var voidType = module.TypeVoid(); + var uintType = module.TypeInt(32, signed: false); + var uvec3Type = module.TypeVector(uintType, 3); + var runtimeArray = module.TypeRuntimeArray(uintType); + module.AddDecoration(runtimeArray, SpirvDecoration.ArrayStride, 4); + var bufferStruct = module.TypeStruct(runtimeArray); + module.AddDecoration(bufferStruct, SpirvDecoration.Block); + module.AddMemberDecoration(bufferStruct, 0, SpirvDecoration.Offset, 0); + var bufferPtrType = module.TypePointer(SpirvStorageClass.StorageBuffer, bufferStruct); + var uintStoragePtr = module.TypePointer(SpirvStorageClass.StorageBuffer, uintType); + + uint MakeBuffer(uint binding, string name) + { + var variable = module.AddGlobalVariable(bufferPtrType, SpirvStorageClass.StorageBuffer); + module.AddName(variable, name); + module.AddDecoration(variable, SpirvDecoration.DescriptorSet, 0); + module.AddDecoration(variable, SpirvDecoration.Binding, binding); + return variable; + } + + var inputVar = MakeBuffer(0, "probe_bounded_input"); + var outputVar = MakeBuffer(1, "probe_output"); + var inputUvec3Ptr = module.TypePointer(SpirvStorageClass.Input, uvec3Type); + var gidVar = module.AddGlobalVariable(inputUvec3Ptr, SpirvStorageClass.Input); + module.AddName(gidVar, "gid"); + module.AddDecoration(gidVar, SpirvDecoration.BuiltIn, (uint)SpirvBuiltIn.GlobalInvocationId); + + var functionType = module.TypeFunction(voidType); + var main = module.BeginFunction(voidType, functionType); + module.AddName(main, "main"); + module.AddLabel(); + var gid = module.AddInstruction(SpirvOp.Load, uvec3Type, gidVar); + var id = module.AddInstruction(SpirvOp.CompositeExtract, uintType, gid, 0); + var sourcePtr = module.AddInstruction( + SpirvOp.AccessChain, uintStoragePtr, inputVar, module.Constant(uintType, 0), id); + var value = module.AddInstruction(SpirvOp.Load, uintType, sourcePtr); + var destinationPtr = module.AddInstruction( + SpirvOp.AccessChain, uintStoragePtr, outputVar, module.Constant(uintType, 0), id); + module.AddStatement(SpirvOp.Store, destinationPtr, value); + module.AddStatement(SpirvOp.Return); + module.EndFunction(); + + module.AddExecutionMode(main, SpirvExecutionMode.LocalSize, 64, 1, 1); + module.AddEntryPoint(SpirvExecutionModel.GLCompute, main, "main", [gidVar, inputVar, outputVar]); + return module.Build(); + } + + public static byte[] CreateHalfConversionProbe() + { + var module = new SpirvModuleBuilder(); + module.AddCapability(SpirvCapability.Shader); + var glsl = module.ImportExtInst("GLSL.std.450"); + + var voidType = module.TypeVoid(); + var uintType = module.TypeInt(32, signed: false); + var floatType = module.TypeFloat(32); + var vec2Type = module.TypeVector(floatType, 2); + var uvec3Type = module.TypeVector(uintType, 3); + var runtimeArray = module.TypeRuntimeArray(uintType); + module.AddDecoration(runtimeArray, SpirvDecoration.ArrayStride, 4); + var bufferStruct = module.TypeStruct(runtimeArray); + module.AddDecoration(bufferStruct, SpirvDecoration.Block); + module.AddMemberDecoration(bufferStruct, 0, SpirvDecoration.Offset, 0); + var bufferPtrType = module.TypePointer(SpirvStorageClass.StorageBuffer, bufferStruct); + var uintStoragePtr = module.TypePointer(SpirvStorageClass.StorageBuffer, uintType); + + uint MakeBuffer(uint binding, string name) + { + var variable = module.AddGlobalVariable(bufferPtrType, SpirvStorageClass.StorageBuffer); + module.AddName(variable, name); + module.AddDecoration(variable, SpirvDecoration.DescriptorSet, 0); + module.AddDecoration(variable, SpirvDecoration.Binding, binding); + return variable; + } + + var inputVar = MakeBuffer(0, "probe_input"); + var outputVar = MakeBuffer(1, "probe_output"); + var inputUvec3Ptr = module.TypePointer(SpirvStorageClass.Input, uvec3Type); + var gidVar = module.AddGlobalVariable(inputUvec3Ptr, SpirvStorageClass.Input); + module.AddName(gidVar, "gid"); + module.AddDecoration(gidVar, SpirvDecoration.BuiltIn, (uint)SpirvBuiltIn.GlobalInvocationId); + + uint UInt(uint value) => module.Constant(uintType, value); + var zeroFloat = module.ConstantFloat(floatType, 0f); + + var functionType = module.TypeFunction(voidType); + var main = module.BeginFunction(voidType, functionType); + module.AddName(main, "main"); + module.AddLabel(); + var gid = module.AddInstruction(SpirvOp.Load, uvec3Type, gidVar); + var id = module.AddInstruction(SpirvOp.CompositeExtract, uintType, gid, 0); + var valuePtr = module.AddInstruction(SpirvOp.AccessChain, uintStoragePtr, inputVar, UInt(0), id); + var value = module.AddInstruction(SpirvOp.Load, uintType, valuePtr); + + var halfBits = module.AddInstruction(SpirvOp.BitwiseAnd, uintType, value, UInt(0xFFFF)); + var unpacked = module.AddInstruction(SpirvOp.ExtInst, vec2Type, glsl, 62, halfBits); + var widened = module.AddInstruction(SpirvOp.CompositeExtract, floatType, unpacked, 0); + var widenedBits = module.AddInstruction(SpirvOp.Bitcast, uintType, widened); + + var asFloat = module.AddInstruction(SpirvOp.Bitcast, floatType, value); + var pair = module.AddInstruction(SpirvOp.CompositeConstruct, vec2Type, asFloat, zeroFloat); + var packed = module.AddInstruction(SpirvOp.ExtInst, uintType, glsl, 58, pair); + var narrowed = module.AddInstruction(SpirvOp.BitwiseAnd, uintType, packed, UInt(0xFFFF)); + + var lowIndex = module.AddInstruction(SpirvOp.IMul, uintType, id, UInt(2)); + var highIndex = module.AddInstruction(SpirvOp.IAdd, uintType, lowIndex, UInt(1)); + var lowPtr = module.AddInstruction(SpirvOp.AccessChain, uintStoragePtr, outputVar, UInt(0), lowIndex); + module.AddStatement(SpirvOp.Store, lowPtr, widenedBits); + var highPtr = module.AddInstruction(SpirvOp.AccessChain, uintStoragePtr, outputVar, UInt(0), highIndex); + module.AddStatement(SpirvOp.Store, highPtr, narrowed); + module.AddStatement(SpirvOp.Return); + module.EndFunction(); + + module.AddExecutionMode(main, SpirvExecutionMode.LocalSize, 64, 1, 1); + module.AddEntryPoint(SpirvExecutionModel.GLCompute, main, "main", [gidVar, inputVar, outputVar]); + return module.Build(); + } + // Read and clear binding 0. Write the count and 64-bit page addresses to binding 1. // Use 64 threads per group, with one thread per 32-page word. public static byte[] CreateFaultBufferProcess() diff --git a/src/SharpEmu.ShaderCompiler.Vulkan/SpirvModuleBuilder.cs b/src/SharpEmu.ShaderCompiler.Vulkan/SpirvModuleBuilder.cs index 0768ec7..6514487 100644 --- a/src/SharpEmu.ShaderCompiler.Vulkan/SpirvModuleBuilder.cs +++ b/src/SharpEmu.ShaderCompiler.Vulkan/SpirvModuleBuilder.cs @@ -185,6 +185,7 @@ public enum SpirvOp : ushort public enum SpirvCapability : uint { Shader = 1, + Tessellation = 3, InterpolationFunction = 52, FragmentBarycentricKhr = 5284, ClipDistance = 32, @@ -192,9 +193,13 @@ public enum SpirvCapability : uint SampledImageArrayDynamicIndexing = 29, StorageImageArrayDynamicIndexing = 31, SampleRateShading = 35, + SampleMaskPostDepthCoverage = 4447, Sampled1D = 43, Image1D = 44, Float16 = 9, + DenormPreserve = 4464, + SignedZeroInfNanPreserve = 4466, + RoundingModeRTE = 4467, Float64 = 10, Int64 = 11, Int64Atomics = 12, @@ -211,9 +216,17 @@ public enum SpirvCapability : uint ShaderLayer = 5253, ShaderViewportIndex = 5254, ShaderViewportIndexLayerExt = 5254, + ShaderNonUniform = 5301, RuntimeDescriptorArray = 5302, + SampledImageArrayNonUniformIndexing = 5307, + StorageImageArrayNonUniformIndexing = 5309, PhysicalStorageBufferAddresses = 5347, - SignedZeroInfNanPreserve = 4466, +} + +public enum SpirvFunctionControl : uint +{ + None = 0, + DontInline = 2, } public enum SpirvStorageClass : uint @@ -234,27 +247,47 @@ public enum SpirvStorageClass : uint public enum SpirvExecutionModel : uint { Vertex = 0, + TessellationControl = 1, + TessellationEvaluation = 2, Fragment = 4, GLCompute = 5, } public enum SpirvExecutionMode : uint { + SpacingEqual = 1, + SpacingFractionalEven = 2, + SpacingFractionalOdd = 3, + VertexOrderCw = 4, + VertexOrderCcw = 5, + PointMode = 10, + Triangles = 22, + Quads = 24, + Isolines = 25, + OutputVertices = 26, OriginUpperLeft = 7, + EarlyFragmentTests = 9, + PostDepthCoverage = 4446, DepthReplacing = 12, LocalSize = 17, + DenormPreserve = 4459, SignedZeroInfNanPreserve = 4461, + RoundingModeRTE = 4462, } public enum SpirvDecoration : uint { + NonUniform = 5300, Block = 2, ArrayStride = 6, BuiltIn = 11, NoPerspective = 13, Flat = 14, + Sample = 17, PerVertexKhr = 5285, + Patch = 15, Location = 30, + Index = 32, Binding = 33, DescriptorSet = 34, Offset = 35, @@ -272,6 +305,12 @@ public enum SpirvBuiltIn : uint ViewportIndex = 10, VertexIndex = 42, InstanceIndex = 43, + PrimitiveId = 7, + InvocationId = 8, + TessLevelOuter = 11, + TessLevelInner = 12, + TessCoord = 13, + PatchVertices = 14, FragCoord = 15, BaryCoordKhr = 5286, BaryCoordNoPerspKhr = 5287, @@ -356,6 +395,7 @@ public sealed class SpirvModuleBuilder private readonly List _annotations = []; private readonly List _typesConstantsGlobals = []; private readonly List _functions = []; + private readonly Dictionary<(uint Width, bool Signed), uint> _integerTypes = []; private readonly Dictionary _floatTypes = []; private readonly Dictionary<(uint Component, uint Count), uint> _vectorTypes = []; @@ -372,7 +412,7 @@ public sealed class SpirvModuleBuilder uint> _imageTypes = []; private readonly Dictionary _sampledImageTypes = []; private readonly Dictionary<(SpirvStorageClass Storage, uint Type), uint> _pointerTypes = []; - private readonly Dictionary<(uint Element, uint Count), uint> _arrayTypes = []; + private readonly Dictionary<(uint Element, uint Count, uint? Stride), uint> _arrayTypes = []; private readonly Dictionary _runtimeArrayTypes = []; private readonly Dictionary _functionTypes = []; private readonly Dictionary<(uint Type, ulong Value), uint> _constants = []; @@ -601,9 +641,12 @@ public sealed class SpirvModuleBuilder return id; } - public uint TypeArray(uint elementType, uint count) + public uint TypeArray(uint elementType, uint count, uint? arrayStride = null) { - var key = (elementType, count); + // Explicit block layout belongs to the type. Sharing it with an array + // in Workgroup storage would also share ArrayStride, which is invalid + // without the explicit workgroup layout extension. + var key = (elementType, count, arrayStride); if (_arrayTypes.TryGetValue(key, out var existing)) { return existing; @@ -612,6 +655,10 @@ public sealed class SpirvModuleBuilder var length = Constant(TypeInt(32, false), count); var id = AllocateId(); Emit(_typesConstantsGlobals, SpirvOp.TypeArray, id, elementType, length); + if (arrayStride is { } stride) + { + AddDecoration(id, SpirvDecoration.ArrayStride, stride); + } _arrayTypes.Add(key, id); return id; } @@ -782,13 +829,25 @@ public sealed class SpirvModuleBuilder return id; } + private readonly List _functionVariables = []; + private int _functionVariableOffset = -1; + public uint BeginFunction(uint returnType, uint functionType) { var id = AllocateId(); + _functionVariableOffset = -1; Emit(_functions, SpirvOp.Function, returnType, id, 0, functionType); return id; } + // Begins a function whose id was allocated earlier, so it can be called before it is emitted. + public uint BeginFunction(uint returnType, uint functionType, uint id, SpirvFunctionControl functionControl = SpirvFunctionControl.None) + { + _functionVariableOffset = -1; + Emit(_functions, SpirvOp.Function, returnType, id, (uint)functionControl, functionType); + return id; + } + public uint AddFunctionParameter(uint type) => EmitResult(_functions, SpirvOp.FunctionParameter, type); @@ -796,6 +855,7 @@ public sealed class SpirvModuleBuilder { var result = id ?? AllocateId(); Emit(_functions, SpirvOp.Label, result); + if (_functionVariableOffset < 0) _functionVariableOffset = _functions.Count; return result; } @@ -805,7 +865,7 @@ public sealed class SpirvModuleBuilder if (initializer.HasValue) { Emit( - _functions, + _functionVariables, SpirvOp.Variable, pointerType, id, @@ -815,7 +875,7 @@ public sealed class SpirvModuleBuilder else { Emit( - _functions, + _functionVariables, SpirvOp.Variable, pointerType, id, @@ -970,7 +1030,20 @@ public sealed class SpirvModuleBuilder public void AddStatement(SpirvOp opcode, params uint[] operands) => Emit(_functions, opcode, operands); - public void EndFunction() => Emit(_functions, SpirvOp.FunctionEnd); + public void EndFunction() + { + // Registers are discovered while emitting the body. SPIR-V requires every + // Function variable to precede the entry block's executable instructions. + if (_functionVariables.Count != 0) + { + if (_functionVariableOffset < 0) + throw new InvalidOperationException("Function variables require an entry block."); + _functions.InsertRange(_functionVariableOffset, _functionVariables); + _functionVariables.Clear(); + } + _functionVariableOffset = -1; + Emit(_functions, SpirvOp.FunctionEnd); + } public byte[] Build() { diff --git a/src/SharpEmu.ShaderCompiler.Vulkan/TilerShaders.cs b/src/SharpEmu.ShaderCompiler.Vulkan/TilerShaders.cs index 3c9e8bc..d25e6ae 100644 --- a/src/SharpEmu.ShaderCompiler.Vulkan/TilerShaders.cs +++ b/src/SharpEmu.ShaderCompiler.Vulkan/TilerShaders.cs @@ -531,6 +531,47 @@ public static class TilerShaders return shaderModule.Finish(main, 64, 1); } + // Packs encoded R/RG bytes into RGBA or extracts them again. No gamma conversion: + // the sRGB image format performs conversion during sampling and attachment writes. + public static byte[] CreateNarrowColorConversion(uint channels, bool widen) + { + if (channels is not (1 or 2)) + throw new ArgumentOutOfRangeException(nameof(channels)); + + var (shaderModule, main, x, y) = BeginConversion(); + var builder = shaderModule.Builder; + ReturnOutsideRectangle(shaderModule, x, y); + var source = shaderModule.Add(shaderModule.LoadArgument(0), + shaderModule.Add(shaderModule.Multiply(y, shaderModule.LoadArgument(6)), + shaderModule.Multiply(x, shaderModule.GetUnsignedConstant(widen ? channels : 4)))); + var destination = shaderModule.Add(shaderModule.LoadArgument(1), + shaderModule.Add(shaderModule.Multiply(y, shaderModule.LoadArgument(7)), + shaderModule.Multiply(x, shaderModule.GetUnsignedConstant(widen ? 4 : channels)))); + if (widen) + { + var value = shaderModule.GetUnsignedConstant(0xff000000); + for (uint channel = 0; channel < channels; channel++) + { + var address = shaderModule.Add(source, shaderModule.GetUnsignedConstant(channel)); + var word = shaderModule.LoadWord(shaderModule.InputBufferVariable, shaderModule.ShiftRight(address, 2)); + var shift = shaderModule.Multiply(shaderModule.And(address, shaderModule.GetUnsignedConstant(3)), shaderModule.GetUnsignedConstant(8)); + var octet = shaderModule.And(builder.AddInstruction(SpirvOp.ShiftRightLogical, shaderModule.UintType, word, shift), shaderModule.GetUnsignedConstant(255)); + value = shaderModule.Or(value, builder.AddInstruction(SpirvOp.ShiftLeftLogical, shaderModule.UintType, octet, shaderModule.GetUnsignedConstant(channel * 8))); + } + builder.AddStatement(SpirvOp.Store, shaderModule.GetWordPointer(shaderModule.OutputBufferVariable, shaderModule.ShiftRight(destination, 2)), value); + } + else + { + // Byte atomics preserve padding and the neighboring packed texels. + for (uint channel = 0; channel < channels; channel++) + CopyElement(shaderModule, shaderModule.Add(source, shaderModule.GetUnsignedConstant(channel)), + shaderModule.Add(destination, shaderModule.GetUnsignedConstant(channel)), 1); + } + builder.AddStatement(SpirvOp.Return); + builder.EndFunction(); + return shaderModule.Finish(main, 64, 1); + } + // Swaps the 16-bit blue and red channels of BGRA16 pixels; the width argument counts pixels. public static byte[] CreateBgra16Swap() { diff --git a/src/SharpEmu.ShaderCompiler/Gen5ShaderIr.cs b/src/SharpEmu.ShaderCompiler/Gen5ShaderIr.cs index 146274a..b4a8fbd 100644 --- a/src/SharpEmu.ShaderCompiler/Gen5ShaderIr.cs +++ b/src/SharpEmu.ShaderCompiler/Gen5ShaderIr.cs @@ -119,6 +119,16 @@ public readonly record struct Gen5ColorComponentMapping } } +// SPI_SHADER_COL_FORMAT selects the representation carried by a compressed EXP. +public enum Gen5PixelExportFormat : byte +{ + Float16 = 4, + Unorm16 = 5, + Snorm16 = 6, + Uint16 = 7, + Sint16 = 8, +} + public readonly record struct Gen5PixelOutputBinding( uint GuestSlot, uint HostLocation, @@ -143,6 +153,12 @@ public readonly record struct Gen5PixelOutputBinding( get => _exportTarget ?? GuestSlot; init => _exportTarget = value; } + + // The fragment output index: 1 is the second source of dual-source blending, which + // shares its location with the first and is not a render target. + public uint Index { get; init; } + + public Gen5PixelExportFormat ExportFormat { get; init; } = Gen5PixelExportFormat.Float16; } public readonly record struct Gen5ComputeSystemRegisters( @@ -255,7 +271,12 @@ public sealed record Gen5BufferMemoryControl( bool Glc, bool Slc, bool Typed = false, - uint TypedFormat = 0) : Gen5InstructionControl; + uint TypedFormat = 0, + bool PackedD16 = false, + uint FormatComponentCount = 0) : Gen5InstructionControl +{ + public uint ComponentCount => FormatComponentCount == 0 ? DwordCount : FormatComponentCount; +} public sealed record Gen5ExportControl( uint Target, @@ -356,6 +377,14 @@ public sealed record Gen5ShaderProgram( public uint ParameterExportMask => _parameterExportMask; + // Fused objects can put their continuation before the entry in guest memory. + // Control-flow PCs remain increasing; S_GETPC still uses the original guest PC. + public IReadOnlyDictionary? InstructionAddressOffsets { get; init; } + public uint? FusedContinuationPc { get; init; } + + public ulong InstructionAddressOffset(uint pc) => + InstructionAddressOffsets is { } offsets && offsets.TryGetValue(pc, out var offset) ? offset : Instructions.FirstOrDefault(instruction => instruction.Pc == pc)?.ProgramOffset ?? pc; + private static uint ComputePixelColorExportMasks( IReadOnlyList instructions) { diff --git a/src/SharpEmu.ShaderCompiler/Gen5ShaderTranslator.cs b/src/SharpEmu.ShaderCompiler/Gen5ShaderTranslator.cs index d3f3f4e..87e53c7 100644 --- a/src/SharpEmu.ShaderCompiler/Gen5ShaderTranslator.cs +++ b/src/SharpEmu.ShaderCompiler/Gen5ShaderTranslator.cs @@ -265,7 +265,7 @@ public static partial class Gen5ShaderTranslator : instruction with { Pc = (uint)rebasedPc, AddressOffset = unchecked(continuationDistance + instruction.ProgramOffset) }); } - program = new Gen5ShaderProgram(entryAddress, instructions); + program = new Gen5ShaderProgram(entryAddress, instructions) { FusedContinuationPc = continuationPc }; error = string.Empty; return true; } @@ -418,8 +418,11 @@ public static partial class Gen5ShaderTranslator return true; } + // S_SWAPPC_B64 with NULL as its link destination discards the return + // address and is a tail transfer, as used by merged LS/HS programs. if (stopAtSetProgramCounter && - string.Equals(name, "SSetpcB64", StringComparison.Ordinal)) + (string.Equals(name, "SSetpcB64", StringComparison.Ordinal) || + (string.Equals(name, "SSwappcB64", StringComparison.Ordinal) && ((word >> 16) & 0x7F) == 125))) { program = new Gen5ShaderProgram(address, instructions); termination = ProgramTermination.SetProgramCounter; @@ -824,6 +827,7 @@ public static partial class Gen5ShaderTranslator 0x09 => "SCbranchExecnz", 0x0A => "SBarrier", 0x0C => "SWaitcnt", + 0x0E => "SSleep", 0x0F => "SSetprio", 0x10 => "SSendmsg", 0x12 => "STrap", @@ -1555,7 +1559,8 @@ public static partial class Gen5ShaderTranslator out uint sizeDwords, out string error) { - var opcode = ((word >> 18) & 0x7F) | ((word & 1) << 7); + // GFX10 MUBUF has eight opcode bits at 18..25; bit zero belongs to OFFSET. + var opcode = (word >> 18) & 0xFF; name = opcode switch { 0x00 => "BufferLoadFormatX", @@ -1588,6 +1593,7 @@ public static partial class Gen5ShaderTranslator 0x23 => "BufferLoadSbyteD16Hi", 0x24 => "BufferLoadShortD16", 0x25 => "BufferLoadShortD16Hi", + 0x27 => "BufferStoreFormatD16HiX", 0x30 => "BufferAtomicSwap", 0x31 => "BufferAtomicCmpswap", 0x32 => "BufferAtomicAdd", @@ -1618,6 +1624,14 @@ public static partial class Gen5ShaderTranslator 0x4C => "BufferAtomicInc", 0x50 => "BufferAtomicSwapX2", 0x5A => "BufferAtomicOrX2", + 0x80 => "BufferLoadFormatD16X", + 0x81 => "BufferLoadFormatD16Xy", + 0x82 => "BufferLoadFormatD16Xyz", + 0x83 => "BufferLoadFormatD16Xyzw", + 0x84 => "BufferStoreFormatD16X", + 0x85 => "BufferStoreFormatD16Xy", + 0x86 => "BufferStoreFormatD16Xyz", + 0x87 => "BufferStoreFormatD16Xyzw", _ => $"MubufRaw{opcode:X2}", }; sizeDwords = (extra >> 24) == 0xFF ? 3u : 2u; @@ -1807,7 +1821,9 @@ public static partial class Gen5ShaderTranslator 0x3F => "ImageSampleCLzO", 0x40 => "ImageGather4", 0x47 => "ImageGather4Lz", + 0x44 => "ImageGather4L", 0x48 => "ImageGather4C", + 0x4C => "ImageGather4CL", 0x4E => "ImageGather4CBCl", 0x4F => "ImageGather4CLz", 0x57 => "ImageGather4LzO", @@ -1863,6 +1879,10 @@ public static partial class Gen5ShaderTranslator return FinishDecode(name, $"unknown-vintrp op=0x{opcode:X1}", out error); } + private static bool HasDoubleVectorResult(string opcode) => opcode is + "VCvtF64I32" or "VCvtF64U32" or "VRcpF64" or "VRsqF64" or + "VSqrtF64" or "VMulF64" or "VFmaF64"; + private static bool FinishDecode(string name, string decodeError, out string error) { error = string.Empty; @@ -2296,9 +2316,12 @@ public static partial class Gen5ShaderTranslator // ordinary vector destination leaves every invocation with a // different value and corrupts scalar addresses derived from // lane data. + var vop1Destination = (word >> 17) & 0xFF; destinations = opcode == "VReadfirstlaneB32" ? [Gen5Operand.Scalar((word >> 17) & 0x7F)] - : [Gen5Operand.Vector((word >> 17) & 0xFF)]; + : HasDoubleVectorResult(opcode) + ? [Gen5Operand.Vector(vop1Destination), Gen5Operand.Vector(vop1Destination + 1)] + : [Gen5Operand.Vector(vop1Destination)]; break; case Gen5ShaderEncoding.Vop2: if (isDpp8) @@ -2427,7 +2450,10 @@ public static partial class Gen5ShaderTranslator Gen5Operand.Source((extra >> 9) & 0x1FF, literal), Gen5Operand.Source((extra >> 18) & 0x1FF, literal), ]; - destinations = [Gen5Operand.Vector(word & 0xFF)]; + var vop3Destination = word & 0xFF; + destinations = HasDoubleVectorResult(opcode) + ? [Gen5Operand.Vector(vop3Destination), Gen5Operand.Vector(vop3Destination + 1)] + : [Gen5Operand.Vector(vop3Destination)]; if (opcode == "VReadlaneB32") { // V_READLANE uses the VOP3A vdst byte even though the @@ -2721,6 +2747,10 @@ public static partial class Gen5ShaderTranslator "BufferStoreFormatXy" => 2u, "BufferStoreFormatXyz" => 3u, "BufferStoreFormatXyzw" => 4u, + "BufferLoadFormatD16X" or "BufferLoadFormatD16Xy" or + "BufferStoreFormatD16X" or "BufferStoreFormatD16Xy" => 1u, + "BufferLoadFormatD16Xyz" or "BufferLoadFormatD16Xyzw" or + "BufferStoreFormatD16Xyz" or "BufferStoreFormatD16Xyzw" => 2u, "BufferLoadUbyte" or "BufferLoadSbyte" or "BufferLoadUshort" or @@ -2766,7 +2796,13 @@ public static partial class Gen5ShaderTranslator ((word >> 13) & 1) != 0, ((word >> 12) & 1) != 0, ((word >> 14) & 1) != 0, - ((extra >> 22) & 1) != 0); + ((extra >> 22) & 1) != 0, + PackedD16: opcode.Contains("FormatD16", StringComparison.Ordinal) && !opcode.Contains("Hi", StringComparison.Ordinal), + FormatComponentCount: opcode.Contains("FormatD16", StringComparison.Ordinal) && !opcode.Contains("Hi", StringComparison.Ordinal) + ? opcode.EndsWith("Xyzw", StringComparison.Ordinal) ? 4u + : opcode.EndsWith("Xyz", StringComparison.Ordinal) ? 3u + : opcode.EndsWith("Xy", StringComparison.Ordinal) ? 2u : 1u + : 0u); break; } case Gen5ShaderEncoding.Mtbuf: @@ -2809,7 +2845,13 @@ public static partial class Gen5ShaderTranslator ((word >> 14) & 1) != 0, ((extra >> 22) & 1) != 0, Typed: true, - TypedFormat: (word >> 19) & 0x7F); + TypedFormat: (word >> 19) & 0x7F, + PackedD16: opcode.Contains("D16", StringComparison.Ordinal), + FormatComponentCount: opcode.Contains("D16", StringComparison.Ordinal) + ? opcode.EndsWith("Xyzw", StringComparison.Ordinal) ? 4u + : opcode.EndsWith("Xyz", StringComparison.Ordinal) ? 3u + : opcode.EndsWith("Xy", StringComparison.Ordinal) ? 2u : 1u + : 0u); break; } case Gen5ShaderEncoding.Mimg: diff --git a/src/SharpEmu.ShaderCompiler/Gen5TessellationData.cs b/src/SharpEmu.ShaderCompiler/Gen5TessellationData.cs new file mode 100644 index 0000000..8ef6cbe --- /dev/null +++ b/src/SharpEmu.ShaderCompiler/Gen5TessellationData.cs @@ -0,0 +1,24 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +namespace SharpEmu.ShaderCompiler; + +// Runtime ABI shared by the merged hull kernel and the native domain pipeline. +public static class Gen5TessellationData +{ + public const uint DwordCount = 19; + public const uint FirstPatch = 0, PatchCount = 1, VertexOffset = 2, InstanceId = 3; + public const uint IndexAddress = 4, IndexSize = 6; + public const uint FactorAddress = 7, FactorBytes = 9; + public const uint OffchipOffset = 10, FactorOffset = 11; + public const uint MinimumLevel = 12, MaximumLevel = 13; + public const uint IndexByteOffset = 14; + // One dispatch runs several hull groups. Each takes its own off-chip slot and its own + // run of the factor ring, as the hardware gives each threadgroup its buffer and offset. + // GroupPatches of zero is a single group in slot zero. + public const uint GroupPatches = 15, OffchipSlotBytes = 16, FactorGroupBytes = 17; + // Patches of one instance. FirstPatch and PatchCount then number the patches of all + // instances in a row, and each thread splits its patch into an instance and a patch + // within it. Zero keeps a single instance. + public const uint PatchesPerInstance = 18; +} diff --git a/src/SharpEmu.ShaderCompiler/Gen5TessellationInfo.cs b/src/SharpEmu.ShaderCompiler/Gen5TessellationInfo.cs new file mode 100644 index 0000000..1c27dc3 --- /dev/null +++ b/src/SharpEmu.ShaderCompiler/Gen5TessellationInfo.cs @@ -0,0 +1,69 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +namespace SharpEmu.ShaderCompiler; + +public enum Gen5TessellationDomain : uint +{ + Isolines, + Triangles, + Quads, +} + +public enum Gen5TessellationSpacing : uint +{ + Equal, + PowerOfTwo, + FractionalOdd, + FractionalEven, +} + +// The merged LS/HS ABI supplies separate vertex and control-point lane counts. +// HullEntryPc names the first HS instruction in the decoded fused program. +public readonly record struct Gen5TessellationHullInfo( + uint InputControlPoints, uint OutputControlPoints, uint PatchesPerGroup, uint HullEntryPc) +{ + public static bool TryDecode(uint configuration, uint hullEntryPc, + out Gen5TessellationHullInfo info, out string error) + { + // VGT_LS_HS_CONFIG stores counts directly, rather than count minus one. + info = new((configuration >> 8) & 0x3F, (configuration >> 14) & 0x3F, + configuration & 0xFF, hullEntryPc); + if (info.InputControlPoints is 0 or > 32 || info.OutputControlPoints is 0 or > 32 || + info.PatchesPerGroup is 0 or > 256 || + info.PatchesPerGroup * Math.Max(info.InputControlPoints, info.OutputControlPoints) > 256) + { + error = "Invalid merged tessellation workgroup configuration."; + return false; + } + error = string.Empty; + return true; + } +} + +// VGT_TF_PARAM describes the fixed-function tessellator, independently of +// where the guest stores control points and tessellation factors. +public readonly record struct Gen5TessellationInfo( + Gen5TessellationDomain Domain, + Gen5TessellationSpacing Spacing, + bool PointMode, + bool Clockwise) +{ + public static bool TryDecode(uint parameter, out Gen5TessellationInfo info, out string error) + { + var domain = parameter & 3; + var spacing = (parameter >> 2) & 7; + var topology = (parameter >> 5) & 7; + info = default; + if (domain > 2 || spacing > 3 || topology > 3 || + (topology == 1 && domain != 0) || (topology >= 2 && domain == 0)) + { + error = $"Invalid tessellator configuration: type={domain} partitioning={spacing} topology={topology}."; + return false; + } + info = new((Gen5TessellationDomain)domain, (Gen5TessellationSpacing)spacing, + topology == 0, topology == 2); + error = string.Empty; + return true; + } +} diff --git a/src/SharpEmu.ShaderCompiler/Gen5TessellationLowering.cs b/src/SharpEmu.ShaderCompiler/Gen5TessellationLowering.cs new file mode 100644 index 0000000..220e611 --- /dev/null +++ b/src/SharpEmu.ShaderCompiler/Gen5TessellationLowering.cs @@ -0,0 +1,59 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +namespace SharpEmu.ShaderCompiler; + +public static class Gen5TessellationLowering +{ + public static uint FindFactorBuffer(Resources.ShaderResourcePlan plan, uint hullEntryPc) + { + uint? resource = null; + foreach (var instruction in plan.Graph.Program.Instructions) + { + // SGPR4 is the factor-ring byte offset in the merged HS ABI. Do + // not mistake a later reuse of that register for a factor store. + if (instruction.Destinations.Any(d => d.Kind == Gen5OperandKind.ScalarRegister && d.Value == 4)) + throw new NotSupportedException("The hull shader overwrites its hardware factor-ring offset."); + if (instruction.Pc < hullEntryPc || !instruction.Opcode.StartsWith("BufferStore", StringComparison.Ordinal) || + instruction.Sources.Count < 3 || instruction.Sources[2] != Gen5Operand.Scalar(4)) continue; + if (!instruction.Opcode.StartsWith("BufferStoreDword", StringComparison.Ordinal) || + instruction.Control is not Gen5BufferMemoryControl { Typed: false } || + plan.Memory.Find(instruction.Pc) is not { Resource: var current } || + current == Resources.MemoryAccessInfo.NoResource) + throw new NotSupportedException("The hull factor ring requires a resolvable raw buffer descriptor."); + if (resource is { } previous && previous != current) + throw new NotSupportedException("The hull shader writes factors to multiple buffer descriptors."); + resource = current; + } + return resource ?? throw new NotSupportedException("The hull shader has no identifiable hardware factor-ring writes."); + } + + // The normalized inputs append the separate HS address after the LS user + // scalars. Move that pair to its architectural s0:s1 before LS can reuse + // those input slots, then publish LS writes before HS reads. Guest + // instruction addresses are preserved for physical-memory reads. + public static Gen5ShaderProgram PrepareMergedHull(Gen5ShaderProgram program, uint hullUserDataRegister) + { + if (hullUserDataRegister < 8 || hullUserDataRegister > 40) + throw new ArgumentOutOfRangeException(nameof(hullUserDataRegister)); + if (program.FusedContinuationPc is not { } continuation || + !program.Instructions.Any(i => i.Pc == continuation)) + throw new ArgumentException("A merged hull shader requires its registered LS/HS continuation.", nameof(program)); + var instructions = new List(program.Instructions.Count + 2) + { + new(0, Gen5ShaderEncoding.Sop1, "SMovB64", [0u], + [Gen5Operand.Scalar(hullUserDataRegister)], [Gen5Operand.Scalar(0)], null), + }; + foreach (var instruction in program.Instructions) + { + if (instruction.Pc == continuation) + instructions.Add(new(continuation + 4, Gen5ShaderEncoding.Sopp, "SBarrier", [0u], [], [], null)); + instructions.Add(instruction with + { + Pc = checked(instruction.Pc + (instruction.Pc < continuation ? 4u : 8u)), + AddressOffset = program.InstructionAddressOffset(instruction.Pc), + }); + } + return new(program.Address, instructions) { FusedContinuationPc = checked(continuation + 8) }; + } +} diff --git a/src/SharpEmu.ShaderCompiler/Ir/Gen5ExecFullAnalysis.cs b/src/SharpEmu.ShaderCompiler/Ir/Gen5ExecFullAnalysis.cs index c37863e..7130d87 100644 --- a/src/SharpEmu.ShaderCompiler/Ir/Gen5ExecFullAnalysis.cs +++ b/src/SharpEmu.ShaderCompiler/Ir/Gen5ExecFullAnalysis.cs @@ -147,7 +147,8 @@ public static class Gen5ExecFullAnalysis } } - var scalarDestination = instruction.Control switch + // GFX10 V_CMPX changes EXEC and ignores the legacy scalar destination. + uint? scalarDestination = opcode.StartsWith("VCmpx", StringComparison.Ordinal) ? null : instruction.Control switch { Gen5Vop3Control control => control.ScalarDestination, Gen5SdwaControl control => control.ScalarDestination, diff --git a/src/SharpEmu.ShaderCompiler/Ir/Gen5ExecRunAnalysis.cs b/src/SharpEmu.ShaderCompiler/Ir/Gen5ExecRunAnalysis.cs new file mode 100644 index 0000000..2d2fb58 --- /dev/null +++ b/src/SharpEmu.ShaderCompiler/Ir/Gen5ExecRunAnalysis.cs @@ -0,0 +1,64 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +namespace SharpEmu.ShaderCompiler.Ir; + +// Which instructions may share one EXEC test. A vector register write is modelled as +// select(EXEC, result, old value), so the old value of every register stays live across every +// write. A run of instructions whose only effect is their vector destination can instead run +// inside one if (EXEC) - the masked-off lanes skip the whole run, which is what the hardware does +// to them anyway, and only the join needs the old values. +// +// A run must stop at anything the wave runs even where this lane is masked off. SALU is not EXEC +// masked on the hardware and every invocation keeps its own copy of the scalar registers, so a +// lane that skipped a scalar write would carry a different value out of the region than its +// neighbours. The same goes for a compare that writes VCC or EXEC (GFX10 keeps that destination in +// the instruction's control, not in its destination list), a cross-lane read, LDS, memory, and the +// DPP encodings, which read a neighbour's register. +public static class Gen5ExecRunAnalysis +{ + public static bool IsExecMaskedVectorAlu(Gen5ShaderInstruction instruction) + { + var opcode = instruction.Opcode; + if (opcode.Length == 0 || opcode[0] != 'V' || + // V_CMP writes a mask register, V_CMPX writes EXEC. + opcode.StartsWith("VCmp", StringComparison.Ordinal) || + // Relative addressing can land on any register. + opcode.StartsWith("VMovrel", StringComparison.Ordinal) || + // Cross-lane reads need the other lanes to have run. + opcode.Contains("Readlane", StringComparison.Ordinal) || + opcode.Contains("Writelane", StringComparison.Ordinal) || + opcode.Contains("Readfirstlane", StringComparison.Ordinal) || + opcode.Contains("Permlane", StringComparison.Ordinal) || + opcode.Contains("Swap", StringComparison.Ordinal) || + // The carry-in/carry-out forms write VCC with no destination operand for it. + opcode.Contains("Co", StringComparison.Ordinal) || + opcode.Contains("DivScale", StringComparison.Ordinal) || + opcode.Contains("DivFmas", StringComparison.Ordinal)) + { + return false; + } + + if (instruction.Destinations.Count == 0) + { + return false; + } + + foreach (var destination in instruction.Destinations) + { + if (destination.Kind != Gen5OperandKind.VectorRegister) + { + return false; + } + } + + return instruction.Control switch + { + null => true, + Gen5Vop3Control control => control.ScalarDestination is null, + Gen5SdwaControl control => control.ScalarDestination is null, + Gen5Vop3pControl => true, + _ => false, + }; + } +} diff --git a/src/SharpEmu.ShaderCompiler/Resources/BindingLayout.cs b/src/SharpEmu.ShaderCompiler/Resources/BindingLayout.cs index 9829b63..66f403a 100644 --- a/src/SharpEmu.ShaderCompiler/Resources/BindingLayout.cs +++ b/src/SharpEmu.ShaderCompiler/Resources/BindingLayout.cs @@ -18,6 +18,8 @@ public enum DescriptorBindingKind : uint FaultBuffer, FlattenedResourceTable, ShaderData, + RuntimeDescriptorTable, + RuntimeDescriptorMisses, Count, } @@ -55,6 +57,8 @@ public static class ImageDescriptorBinding private const uint SampledCompare2DBinding = 43; private const uint SampledCompare2DArrayBinding = 44; private const uint SampledCompareCubeBinding = 45; + private const uint StorageSintBinding = 46; + private const uint StorageCubeSintBinding = 51; public static DescriptorBindingKind? ForImage(ImageResource image) { @@ -115,6 +119,7 @@ public static class ImageDescriptorBinding { ImageNumericClass.Float => (DescriptorBindingKind)StorageCubeFloatBinding, ImageNumericClass.Uint => (DescriptorBindingKind)StorageCubeUintBinding, + ImageNumericClass.Sint => (DescriptorBindingKind)StorageCubeSintBinding, _ => null, }; } @@ -168,6 +173,9 @@ public static class ImageDescriptorBinding case ImageNumericClass.Uint: baseBinding = StorageUintBinding; break; + case ImageNumericClass.Sint: + baseBinding = StorageSintBinding; + break; default: return null; } @@ -249,7 +257,7 @@ public static class ImageDescriptorBinding public static uint ArrayIndex(DescriptorBindingKind kind) => (uint)kind - BindingLayout.FirstImageBinding; public static bool IsCube(DescriptorBindingKind kind) => - (uint)kind is >= SampledCubeFloatBinding and <= AtomicCubeUintBinding or SampledCompareCubeBinding; + (uint)kind is >= SampledCubeFloatBinding and <= AtomicCubeUintBinding or SampledCompareCubeBinding or StorageCubeSintBinding; private static readonly ImageDimension[] SampledDimensions = [ @@ -311,7 +319,12 @@ public static class ImageDescriptorBinding return (ImageResourceClass.Storage, offset / 5 == 0 ? ImageNumericClass.Float : ImageNumericClass.Uint, StorageDimensions[offset % 5], false); } - if (index >= AtomicUintBinding && index < (uint)DescriptorBindingKind.Samplers) + if (index >= StorageSintBinding && index < StorageCubeSintBinding) + return (ImageResourceClass.Storage, ImageNumericClass.Sint, StorageDimensions[index - StorageSintBinding], false); + if (index == StorageCubeSintBinding) + return (ImageResourceClass.Storage, ImageNumericClass.Sint, ImageDimension.Dim2DArray, false); + + if (index >= AtomicUintBinding && index < SampledCubeFloatBinding) { return (ImageResourceClass.Storage, ImageNumericClass.Uint, StorageDimensions[index - AtomicUintBinding], true); } @@ -326,7 +339,14 @@ public sealed class BindingLayout : IEquatable { public const uint FirstImageBinding = 1; public const uint FirstStorageImageBinding = 22; - public const uint ImageBindingCount = 45; + public const uint ImageBindingCount = 51; + // A bindless shader aliases all sampled image classes through one descriptor + // binding and all storage classes through another. The local slot table still + // keeps the guest image classes distinct. + public const uint BindlessSampledImageBinding = 0; + public const uint BindlessStorageImageBinding = 1; + // The persistent heap's sampler array, which runtime-descriptor accesses index. + public const uint BindlessSamplerBinding = 2; public const uint NoShaderBase = uint.MaxValue; public const uint ShaderBaseDwordCount = 2; private const int ScalarRegisterCount = 256; @@ -337,12 +357,50 @@ public sealed class BindingLayout : IEquatable public uint MemoryOffsetDword { get; init; } public uint MemoryOffsetCount { get; init; } public bool UsesDispatchThreadLimits { get; init; } + public bool UsesTessellationData { get; init; } + + // The host writes each buffer's descriptor stride into shader data (two 16-bit halves + // per dword) and the shader indexes with it, so the stride is not compiled in: one + // program serves every stride a title binds instead of one compile per stride. + public bool UsesRuntimeBufferStrides { get; init; } + // Vulkan can put image arrays in one persistent descriptor set. The per-draw + // flattened table then starts with the local-image -> heap-slot mapping. + public bool UsesBindlessImages { get; init; } + + // The module reads the internal-resolution dwords: the factor that maps a guest pixel + // onto host texels, its reciprocal, the mask of scaled image resources and the factor + // the pixel position is divided by. + public bool UsesRenderScale { get; init; } public IReadOnlyList UserDataRegisters { get; init; } = []; public IReadOnlyList Descriptors { get; init; } = []; - public uint DispatchThreadLimitsDword => MemoryOffsetDword + (MemoryOffsetCount + 3) / 4; + public const uint RenderScaleDwordCount = 5; - public uint ShaderDataDwordCount => DispatchThreadLimitsDword + (UsesDispatchThreadLimits ? 3u : 0u); + // Two entries per dword: a 14-bit stride followed by a 2-bit count of native + // tail-padding bytes. The shader subtracts padding for guest byte bounds. + public uint BufferStrideDword => MemoryOffsetDword + (MemoryOffsetCount + 3) / 4; + + public uint BufferStrideDwordCount => UsesRuntimeBufferStrides ? (MemoryOffsetCount + 1) / 2 : 0; + + public uint DispatchThreadLimitsDword => BufferStrideDword + BufferStrideDwordCount; + + public uint TessellationDataDword => DispatchThreadLimitsDword + (UsesDispatchThreadLimits ? 3u : 0u); + public uint RenderScaleDword => TessellationDataDword + (UsesTessellationData ? Gen5TessellationData.DwordCount : 0u); + + public uint ShaderDataDwordCount => RenderScaleDword + (UsesRenderScale ? RenderScaleDwordCount : 0u); + + public static uint ImageSlotTableDwordCount(ShaderResourceInfo info) + { + var count = 0u; + foreach (var image in info.Images) + { + _ = ImageDescriptorBinding.ForImage(image) ?? throw new ResourcePlanException( + "image slot table cannot classify an image without a descriptor binding"); + count += image.MipMode == ImageMipMode.DynamicStorage ? image.MipCount : 1u; + } + + return count; + } public bool UsesPushData => PushDataStartDword != PushData.NoStart; @@ -561,13 +619,22 @@ public sealed class BindingLayout : IEquatable bool usesFlattenedTable, bool usesShaderBase, uint pushDataStartDword = 0, - bool usesDispatchThreadLimits = false) + bool usesDispatchThreadLimits = false, + bool usesRenderScale = false, + bool usesBindlessImages = false, + bool usesRuntimeBufferStrides = false, + bool usesTessellationData = false) { var shaderBaseDword = usesShaderBase ? (uint)userDataRegisters.Count : NoShaderBase; var memoryOffsetDword = (uint)userDataRegisters.Count + (usesShaderBase ? ShaderBaseDwordCount : 0); var memoryOffsetCount = (uint)info.Buffers.Count; - var shaderDataDwords = memoryOffsetDword + (memoryOffsetCount + 3) / 4 + (usesDispatchThreadLimits ? 3u : 0u); - var pushStart = PushData.StartFor(pushDataStartDword, shaderDataDwords); + usesRuntimeBufferStrides &= memoryOffsetCount != 0; + var shaderDataDwords = memoryOffsetDword + (memoryOffsetCount + 3) / 4 + + (usesRuntimeBufferStrides ? (memoryOffsetCount + 1) / 2 : 0u) + (usesDispatchThreadLimits ? 3u : 0u) + + (usesTessellationData ? Gen5TessellationData.DwordCount : 0u) + (usesRenderScale ? RenderScaleDwordCount : 0u); + // The fixed-function control bridge shares the domain stage's runtime + // buffer. Keep that ABI independent of per-stage push-data packing. + var pushStart = usesTessellationData ? PushData.NoStart : PushData.StartFor(pushDataStartDword, shaderDataDwords); var descriptors = new List(); if (info.Buffers.Count != 0) { @@ -619,13 +686,25 @@ public sealed class BindingLayout : IEquatable descriptors.Add(new DescriptorBinding(DescriptorBindingKind.GlobalDataShare, [])); } + if (info.UsesRuntimeDescriptors) + { + if (!usesBindlessImages) + { + throw new ResourcePlanException("runtime image descriptors need the persistent bindless heap"); + } + + descriptors.Add(new DescriptorBinding(DescriptorBindingKind.RuntimeDescriptorTable, [])); + descriptors.Add(new DescriptorBinding(DescriptorBindingKind.RuntimeDescriptorMisses, [])); + } + if (info.UsesDeviceAddresses) { descriptors.Add(new DescriptorBinding(DescriptorBindingKind.DeviceAddressPageTable, [])); descriptors.Add(new DescriptorBinding(DescriptorBindingKind.FaultBuffer, [])); } - var flattened = usesFlattenedTable || info.Images.Any(image => image.IndirectSearchIterations != 0); + var flattened = usesFlattenedTable || info.Images.Any(image => image.IndirectSearchIterations != 0) || + (usesBindlessImages && info.Images.Count != 0); if (flattened) { descriptors.Add(new DescriptorBinding(DescriptorBindingKind.FlattenedResourceTable, [])); @@ -644,6 +723,13 @@ public sealed class BindingLayout : IEquatable MemoryOffsetDword = memoryOffsetDword, MemoryOffsetCount = memoryOffsetCount, UsesDispatchThreadLimits = usesDispatchThreadLimits, + UsesTessellationData = usesTessellationData, + UsesRuntimeBufferStrides = usesRuntimeBufferStrides, + // The set split is a pipeline-wide ABI. A stage without images must + // still place its buffers on set 1 when another stage uses set 0 for + // the persistent image heap. + UsesBindlessImages = usesBindlessImages, + UsesRenderScale = usesRenderScale, UserDataRegisters = userDataRegisters, Descriptors = descriptors, }; @@ -657,13 +743,17 @@ public sealed class BindingLayout : IEquatable MemoryOffsetDword == other.MemoryOffsetDword && MemoryOffsetCount == other.MemoryOffsetCount && UsesDispatchThreadLimits == other.UsesDispatchThreadLimits && + UsesTessellationData == other.UsesTessellationData && + UsesRuntimeBufferStrides == other.UsesRuntimeBufferStrides && + UsesBindlessImages == other.UsesBindlessImages && + UsesRenderScale == other.UsesRenderScale && UserDataRegisters.SequenceEqual(other.UserDataRegisters) && Descriptors.Count == other.Descriptors.Count && Descriptors.Zip(other.Descriptors).All(pair => pair.First.Kind == pair.Second.Kind && pair.First.Resources.SequenceEqual(pair.Second.Resources)); public override bool Equals(object? obj) => Equals(obj as BindingLayout); - public override int GetHashCode() => HashCode.Combine(PushDataStartDword, MemoryOffsetDword, MemoryOffsetCount, Descriptors.Count, UsesDispatchThreadLimits); + public override int GetHashCode() => HashCode.Combine(PushDataStartDword, MemoryOffsetDword, MemoryOffsetCount, Descriptors.Count, UsesDispatchThreadLimits, UsesRenderScale); } // Recomputes the layout an emitter was given from the same inputs and the same push @@ -685,7 +775,7 @@ public static class BindingLayoutValidator throw new ResourcePlanException("Only a compute shader can use dispatch thread limits."); } - var expected = BindingLayout.Allocate(info, userDataRegisters, usesGlobalDataShare, usesFlattenedTable, usesShaderBase, layout.AllocationCursor, layout.UsesDispatchThreadLimits); + var expected = BindingLayout.Allocate(info, userDataRegisters, usesGlobalDataShare, usesFlattenedTable, usesShaderBase, layout.AllocationCursor, layout.UsesDispatchThreadLimits, layout.UsesRenderScale, layout.UsesBindlessImages, layout.UsesRuntimeBufferStrides, layout.UsesTessellationData); if (!expected.Equals(layout)) { throw new ResourcePlanException( @@ -696,6 +786,7 @@ public static class BindingLayoutValidator private static string Describe(BindingLayout layout) => $"push={layout.PushDataStartDword} base={layout.ShaderBaseDword} offsets={layout.MemoryOffsetDword}+{layout.MemoryOffsetCount} " + + $"strides={layout.UsesRuntimeBufferStrides} " + $"user=[{string.Join(",", layout.UserDataRegisters)}] " + string.Join(" ", layout.Descriptors.Select(binding => $"{binding.Kind}:{string.Join(",", binding.Resources)}")); } diff --git a/src/SharpEmu.ShaderCompiler/Resources/BufferCandidateTablePlanner.cs b/src/SharpEmu.ShaderCompiler/Resources/BufferCandidateTablePlanner.cs index 993360a..170a0e3 100644 --- a/src/SharpEmu.ShaderCompiler/Resources/BufferCandidateTablePlanner.cs +++ b/src/SharpEmu.ShaderCompiler/Resources/BufferCandidateTablePlanner.cs @@ -25,6 +25,7 @@ internal static class BufferCandidateTablePlanner // with the component immediates 0,4,8,12 that one dwordx4 read produces. ScalarValue? srt = null; ScalarValue? dynamicOffset = null; + uint descriptorOffset = 0; for (var word = 0; word < 4; word++) { var read = handle.Operands[word]; @@ -35,11 +36,14 @@ internal static class BufferCandidateTablePlanner var memory = plan.Memory[read.MemoryIndex]; if (memory.Kind != MemoryResourceKind.ScalarBuffer || memory.DataBits != 32 || memory.DataDwords != 1 || - memory.Offset != (uint)word * sizeof(uint)) + (word != 0 && memory.Offset != descriptorOffset + (uint)word * sizeof(uint))) { return false; } + if (word == 0) + descriptorOffset = memory.Offset; + if (srt is null) { srt = read.Operands[0]; @@ -61,17 +65,21 @@ internal static class BufferCandidateTablePlanner // offset = base + index * stride var offset = dynamicOffset!; - var baseOffset = 0u; + var baseOffset = descriptorOffset; if (offset.Kind == ScalarValueKind.Operation && offset.Operation == ScalarOperation.IAdd32) { if (offset.Operands[0].IsConstant) { - baseOffset = offset.Operands[0].ConstantU32; + if ((ulong)baseOffset + offset.Operands[0].ConstantU32 > uint.MaxValue) + return false; + baseOffset += offset.Operands[0].ConstantU32; offset = offset.Operands[1]; } else if (offset.Operands[1].IsConstant) { - baseOffset = offset.Operands[1].ConstantU32; + if ((ulong)baseOffset + offset.Operands[1].ConstantU32 > uint.MaxValue) + return false; + baseOffset += offset.Operands[1].ConstantU32; offset = offset.Operands[0]; } } @@ -114,7 +122,10 @@ internal static class BufferCandidateTablePlanner } var srtExtent = SrtByteExtent(plan, srt!); - var minOffset = unchecked(baseOffset + initial.ConstantU32 * stride); + var firstOffset = (ulong)baseOffset + (ulong)initial.ConstantU32 * stride; + if (firstOffset > uint.MaxValue) + return false; + var minOffset = (uint)firstOffset; var candidate = new BufferCandidateTablePlan { SrtHandle = srt!, @@ -145,7 +156,10 @@ internal static class BufferCandidateTablePlanner return false; } - maxOffset = unchecked(candidate.StaticCandidateOffset(count - 1) + BufferCandidateTablePlan.DescriptorByteSize); + var endOffset = firstOffset + (ulong)(count - 1) * candidate.CandidateSpacing + BufferCandidateTablePlan.DescriptorByteSize; + if (endOffset > uint.MaxValue) + return false; + maxOffset = (uint)endOffset; } else { diff --git a/src/SharpEmu.ShaderCompiler/Resources/GuestImageFormat.cs b/src/SharpEmu.ShaderCompiler/Resources/GuestImageFormat.cs index 56883f7..0303cdd 100644 --- a/src/SharpEmu.ShaderCompiler/Resources/GuestImageFormat.cs +++ b/src/SharpEmu.ShaderCompiler/Resources/GuestImageFormat.cs @@ -34,7 +34,7 @@ public static class GuestImageFormat // is unsupported for sampling. private static readonly (uint Format, ImageNumericClass Class)[] SampledFormats = [ - (1, ImageNumericClass.Float), (Format8Uscaled, ImageNumericClass.Float), (5, ImageNumericClass.Uint), (7, ImageNumericClass.Float), (8, ImageNumericClass.Float), + (1, ImageNumericClass.Float), (Format8Uscaled, ImageNumericClass.Float), (5, ImageNumericClass.Uint), (6, ImageNumericClass.Sint), (7, ImageNumericClass.Float), (8, ImageNumericClass.Float), (11, ImageNumericClass.Uint), (12, ImageNumericClass.Sint), (13, ImageNumericClass.Float), (14, ImageNumericClass.Float), (15, ImageNumericClass.Float), (16, ImageNumericClass.Float), (18, ImageNumericClass.Uint), (19, ImageNumericClass.Sint), (20, ImageNumericClass.Uint), (21, ImageNumericClass.Sint), (22, ImageNumericClass.Float), (23, ImageNumericClass.Float), (24, ImageNumericClass.Float), diff --git a/src/SharpEmu.ShaderCompiler/Resources/MemoryAccessInfo.cs b/src/SharpEmu.ShaderCompiler/Resources/MemoryAccessInfo.cs index 67ff6fe..515e6e1 100644 --- a/src/SharpEmu.ShaderCompiler/Resources/MemoryAccessInfo.cs +++ b/src/SharpEmu.ShaderCompiler/Resources/MemoryAccessInfo.cs @@ -112,6 +112,11 @@ public sealed class MemoryAccessInfo // through the descriptor in its registers instead of a host-bound buffer. public bool DeviceDescriptor { get; set; } + // A sampled image access whose image and sampler descriptors have no plan-time source + // (a waterfall loop, a per-pixel material pointer): the shader looks the words in its + // registers up in the host's global table (RuntimeDescriptorTable). + public bool RuntimeDescriptor { get; set; } + // The guest V# behind a buffer access, filled in by the resource tracker. // Null until then. Its provenance tells the backend which lowering // strategy to pick; the access shape (Typed/Formatted/Access) completes @@ -134,7 +139,8 @@ public sealed class MemoryAccessInfo ImageHasMip == other.ImageHasMip && ImageR128 == other.ImageR128 && Glc == other.Glc && Slc == other.Slc && IndexEnabled == other.IndexEnabled && OffsetEnabled == other.OffsetEnabled && Resource == other.Resource && Sampler == other.Sampler && PlanningOnly == other.PlanningOnly && - DeviceDescriptor == other.DeviceDescriptor && AddressSpace == other.AddressSpace; + DeviceDescriptor == other.DeviceDescriptor && AddressSpace == other.AddressSpace && + RuntimeDescriptor == other.RuntimeDescriptor; } // Every memory access of a program, indexed by program counter and component. @@ -247,7 +253,7 @@ public sealed class MemoryAccessTable Offset = unchecked((uint)control.OffsetBytes), DataDwords = Math.Max(control.DwordCount, 1u), DataBits = bits, - ComponentCount = Math.Max(control.DwordCount, 1u), + ComponentCount = Math.Max(control.ComponentCount, 1u), DataFormat = dataFormat, NumberFormat = numberFormat, DataSigned = signed, diff --git a/src/SharpEmu.ShaderCompiler/Resources/ResourceMaterializationCache.cs b/src/SharpEmu.ShaderCompiler/Resources/ResourceMaterializationCache.cs index 759e5bc..91f668b 100644 --- a/src/SharpEmu.ShaderCompiler/Resources/ResourceMaterializationCache.cs +++ b/src/SharpEmu.ShaderCompiler/Resources/ResourceMaterializationCache.cs @@ -37,6 +37,7 @@ public sealed class ResourceMaterializationCache private static long _totalStale; private static long _totalStaleUnreadable; private static long _totalRefreshes; + private static long _totalBypassed; [ThreadStatic] private static bool _readingTable; @@ -47,6 +48,11 @@ public sealed class ResourceMaterializationCache private set => _readingTable = value; } + private const int ProbationLookups = 64; + // A plan keeps using the cache while at least one lookup in this many hits. + private const int MinimumHitShare = 8; + + public long Bypassed { get; private set; } public long Hits { get; private set; } public long Misses { get; private set; } public long Uncacheable { get; private set; } @@ -61,9 +67,10 @@ public sealed class ResourceMaterializationCache var stale = Interlocked.Exchange(ref _totalStale, 0); var staleUnreadable = Interlocked.Exchange(ref _totalStaleUnreadable, 0); var refreshes = Interlocked.Exchange(ref _totalRefreshes, 0); + var bypassed = Interlocked.Exchange(ref _totalBypassed, 0); var total = hits + misses; return FormattableString.Invariant( - $"[PERF][RESOURCE_CACHE] hits={hits} misses={misses} stale={stale} stale_unreadable={staleUnreadable} refreshes={refreshes} uncacheable={uncacheable} hit_rate={(total == 0 ? 0 : hits * 100.0 / total):F1}% {RawReadPrefetch.TakeReport()}"); + $"[PERF][RESOURCE_CACHE] hits={hits} misses={misses} stale={stale} stale_unreadable={staleUnreadable} refreshes={refreshes} bypassed={bypassed} uncacheable={uncacheable} hit_rate={(total == 0 ? 0 : hits * 100.0 / total):F1}% {RawReadPrefetch.TakeReport()}"); } public bool Materialize( @@ -74,6 +81,17 @@ public sealed class ResourceMaterializationCache ref ResourceSpecialization specialization, out ResourceMaterializationFailure failure) { + // Most draws of a plan whose user data points at per-draw constants never repeat their + // inputs; recording their reads and storing entries then costs more than a plain walk. + // Such a plan materializes directly and probes the cache again on every 64th draw. + var lookups = ++plan.CacheLookups; + if (lookups > ProbationLookups && plan.CacheHits * MinimumHitShare < lookups && (lookups & 63) != 0) + { + Bypassed++; + Interlocked.Increment(ref _totalBypassed); + return ResourceMaterializer.Materialize(plan, inputs, ref snapshot, ref specialization, out failure); + } + var key = KeyOf(plan, inputs); var found = TryFind(key, plan, inputs, out var cached); if (found) @@ -94,6 +112,7 @@ public sealed class ResourceMaterializationCache } Hits++; + plan.CacheHits++; Interlocked.Increment(ref _totalHits); snapshot = variant.Snapshot; specialization = variant.Specialization; @@ -107,6 +126,7 @@ public sealed class ResourceMaterializationCache if (TryRefreshTable(key, cached, plan, inputs, residentReader, out var refreshed)) { TableRefreshes++; + plan.CacheHits++; Interlocked.Increment(ref _totalRefreshes); snapshot = refreshed.Snapshot; specialization = refreshed.Specialization; @@ -130,6 +150,7 @@ public sealed class ResourceMaterializationCache ShaderBase = inputs.ShaderBase, ReadMemory = recorder.Wrap(inputs.ReadMemory, clean: false), ReadCleanMemory = recorder.Wrap(inputs.ReadCleanMemory, clean: true), + ReadCleanWords = recorder.Wrap(inputs.ReadCleanWords), ReadResidentMemory = recorder.WrapResident(inputs.ReadResidentMemory), ReadsClean = inputs.ReadsClean, ComputeState = inputs.ComputeState, @@ -211,6 +232,7 @@ public sealed class ResourceMaterializationCache ShaderBase = inputs.ShaderBase, ReadMemory = recorder.Wrap(inputs.ReadMemory, clean: false), ReadCleanMemory = recorder.Wrap(inputs.ReadCleanMemory, clean: true), + ReadCleanWords = recorder.Wrap(inputs.ReadCleanWords), ReadResidentMemory = recorder.WrapResident(inputs.ReadResidentMemory), ReadsClean = inputs.ReadsClean, ComputeState = inputs.ComputeState, @@ -415,6 +437,7 @@ public sealed class ResourceMaterializationCache private sealed class ReadRecorder { private readonly List<(ulong Address, uint Word, bool Clean, bool Table)> _reads = new(); + private readonly Dictionary _readIndex = new(); private bool _inTable; private GuestWordReader? _reader; private GuestWordReader? _cleanReader; @@ -438,6 +461,7 @@ public sealed class ResourceMaterializationCache public void Reset() { _reads.Clear(); + _readIndex.Clear(); // Retain ordinary descriptor walks without keeping unusually large // tables alive for the lifetime of the renderer. if (_reads.Capacity > 16384) @@ -475,7 +499,7 @@ public sealed class ResourceMaterializationCache Failed = true; return false; } - _reads.Add((address, word, clean, _inTable)); + Record(address, word, clean); return true; } @@ -493,11 +517,40 @@ public sealed class ResourceMaterializationCache return false; for (var offset = 0; offset + sizeof(uint) <= destination.Length; offset += sizeof(uint)) - _reads.Add((address + (ulong)offset, - System.Buffers.Binary.BinaryPrimitives.ReadUInt32LittleEndian(destination[offset..]), clean, _inTable)); + Record(address + (ulong)offset, + System.Buffers.Binary.BinaryPrimitives.ReadUInt32LittleEndian(destination[offset..]), clean); return true; } + public GuestWordsReader? Wrap(GuestWordsReader? inner) + { + if (inner is null) return null; + return (ulong address, Span words) => + { + // A refused range falls back to individual reads; only those + // determine whether the materialization is uncacheable. + if (!inner(address, words)) return false; + for (var index = 0; index < words.Length; index++) + Record(address + (ulong)index * sizeof(uint), words[index], true); + return true; + }; + } + + private void Record(ulong address, uint word, bool clean) + { + if (_readIndex.TryGetValue(address, out var index)) + { + var previous = _reads[index]; + // A changing input during one materialization cannot be represented + // by a cache entry that validates only one value for this address. + if (previous.Word != word) Failed = true; + _reads[index] = (address, previous.Word, previous.Clean || clean, previous.Table && _inTable); + return; + } + _readIndex.Add(address, _reads.Count); + _reads.Add((address, word, clean, _inTable)); + } + public Entry Build(ShaderResourcePlan plan, ResourceRuntimeInputs inputs, ResourceSnapshot snapshot, ResourceSpecialization specialization) { diff --git a/src/SharpEmu.ShaderCompiler/Resources/ResourceMaterializer.cs b/src/SharpEmu.ShaderCompiler/Resources/ResourceMaterializer.cs index b65eb7e..0bfe0ff 100644 --- a/src/SharpEmu.ShaderCompiler/Resources/ResourceMaterializer.cs +++ b/src/SharpEmu.ShaderCompiler/Resources/ResourceMaterializer.cs @@ -14,6 +14,12 @@ public static class ResourceMaterializer private const ulong AddressMask = 0x0000_FFFF_FFFF_FFFFul; private const ulong MaxIndirectImageProbes = 65536; + // Why the last Materialize call on this thread returned false, for the host's fatal. + [ThreadStatic] + private static string? _lastFailureDetail; + + public static string? LastFailureDetail => _lastFailureDetail; + // Written to standard error like every specialization refusal; the host turns it // into its fatal. public static Action SpecializationFailed { get; set; } = message => Console.Error.WriteLine($"shader resource specialization failed: {message}"); @@ -69,6 +75,7 @@ public static class ResourceMaterializer Action? captureIndirectImageFailure = null) { using var totalProfile = ResourceMaterializationProfile.Measure(ResourceMaterializationProfile.Phase.Total); + _lastFailureDetail = null; if (!MaterializeSnapshot(plan, inputs, captureIndirectImageFailure is not null, out var materialized, out failure)) { return false; @@ -127,17 +134,31 @@ public static class ResourceMaterializer snapshot = new MaterializedSnapshot(); if (plan.RequiresSpecializationMemory && inputs.ReadCleanMemory is null) { - return false; + return Detail("the plan reads guest memory for specialization but no clean-memory reader is available"); } if (!RuntimeValueEvaluator.EvaluateSources(plan, plan.MaterializationSources, inputs, plan.CleanFlatSlots, evaluateTable: true, out var values, out var table, out var activeSources, additionalTableWords: checked(plan.WrittenRangeCount * ShaderResourcePlan.WrittenRangeDwordCount))) { - SpecializationFailed(DiagnoseSnapshotEvaluationFailure(plan, inputs, activeSources)); + var diagnosis = DiagnoseSnapshotEvaluationFailure(plan, inputs, activeSources); + _lastFailureDetail = diagnosis; + SpecializationFailed(diagnosis); return false; } + foreach (var source in plan.DescriptorSources) + { + if (source.ZeroExtentBufferSource is not { } bufferSource) continue; + if (!RuntimeValueEvaluator.EvaluateSources(plan, [bufferSource], inputs.WithReader(inputs.ReadCleanMemory), [], + evaluateTable: false, out var descriptors, out _) || + descriptors.Count != 1 || descriptors[0].DwordCount != 4 || ScalarBufferSize(descriptors[0].Dwords) != 0) + { + SpecializationFailed("a dynamically loaded descriptor requires an empty source buffer"); + return false; + } + } + var cursor = 0; snapshot.Buffers = new uint[plan.Info.Buffers.Count][]; for (var index = 0; index < snapshot.Buffers.Length; index++) @@ -158,10 +179,11 @@ public static class ResourceMaterializer var cleanInputs = inputs.WithReader(inputs.ReadCleanMemory); if (indirect.DirectCandidates is { } directCandidates) { - if (!RuntimeValueEvaluator.EvaluateSources(plan, directCandidates.Select(candidate => candidate.Source).ToArray(), + var candidateSources = directCandidates.Select(candidate => candidate.Source).ToArray(); + if (!RuntimeValueEvaluator.EvaluateSources(plan, candidateSources, cleanInputs, [], evaluateTable: false, out var descriptors, out _)) { - return false; + return Detail($"image {imageIndex} direct candidates: " + DiagnoseSources(plan, cleanInputs, candidateSources)); } var directTable = new IndirectImageTable { Resource = (uint)imageIndex }; for (var candidateIndex = 0; candidateIndex < descriptors.Count; candidateIndex++) @@ -186,7 +208,7 @@ public static class ResourceMaterializer { if (!RuntimeValueEvaluator.EvaluateSources(plan, [indirect.HeapSource], cleanInputs, [], evaluateTable: false, out var waveSources, out _)) - return false; + return Detail($"image {imageIndex} wave-indexed heap source: " + DiagnoseSources(plan, cleanInputs, [indirect.HeapSource])); if (!MaterializeWaveIndexedImage(indirect, waveIndexed, waveSources[0], image, image.R128, inputs, out var waveTable, out failure)) return false; snapshot.Images[imageIndex] = waveTable.Descriptors[(int)waveTable.Candidates[0]].Dwords; @@ -201,7 +223,7 @@ public static class ResourceMaterializer { if (!RuntimeValueEvaluator.EvaluateSources(plan, [indirect.HeapSource], cleanInputs, [], evaluateTable: false, out var denseSources, out _)) - return false; + return Detail($"image {imageIndex} dense heap source: " + DiagnoseSources(plan, cleanInputs, [indirect.HeapSource])); if (!MaterializeDenseIndirectImage(indirect, denseSources[0], image, image.R128, inputs, out var denseTable, out failure)) return false; snapshot.Images[imageIndex] = denseTable.Descriptors[(int)denseTable.Candidates[0]].Dwords; @@ -214,7 +236,8 @@ public static class ResourceMaterializer } if (!RuntimeValueEvaluator.EvaluateSources(plan, [indirect.MaterialSource, indirect.HeapSource], cleanInputs, [], evaluateTable: false, out var tables, out _)) { - return false; + return Detail($"image {imageIndex} material/heap sources: " + + DiagnoseSources(plan, cleanInputs, [indirect.MaterialSource, indirect.HeapSource])); } if (!MaterializeIndirectImage(plan, indirect, tables[0], tables[1], image, inputs, @@ -244,22 +267,38 @@ public static class ResourceMaterializer snapshot.Samplers = new uint[plan.Info.Samplers.Count][]; for (var index = 0; index < snapshot.Samplers.Length; index++) - snapshot.Samplers[index] = values[cursor++].Dwords; + { + var samplerSource = plan.DescriptorSources[(int)plan.Info.Samplers[index].Source]; + var samplerWords = values[cursor++]; + if (samplerSource.PointerTable is { } pointerTable) + { + if (!MaterializePointerTableSampler(pointerTable, samplerWords, inputs, out var sampler)) + return _lastFailureDetail is not null ? false : Detail($"sampler {index} pointer table failed"); + snapshot.Samplers[index] = sampler; + continue; + } + + snapshot.Samplers[index] = samplerWords.Dwords; + } // Bounded runtime V# tables: the whole table is read and validated before it is // published, so a single unreadable candidate leaves the previous snapshot intact. foreach (var candidatePlan in plan.BufferCandidateTables) { - if (candidatePlan.SourceSrtResource == DescriptorConstants.NoIndex || - !RuntimeValueEvaluator.EvaluateSources(plan, [candidatePlan.SourceSrtResource], inputs, [], evaluateTable: false, + if (candidatePlan.SourceSrtResource == DescriptorConstants.NoIndex) + { + return Detail("buffer candidate table has no SRT source resource"); + } + + if (!RuntimeValueEvaluator.EvaluateSources(plan, [candidatePlan.SourceSrtResource], inputs, [], evaluateTable: false, out var srtSources, out _) || srtSources.Count == 0) { - return false; + return Detail("buffer candidate table SRT source: " + DiagnoseSources(plan, inputs, [candidatePlan.SourceSrtResource])); } if (!MaterializeBufferCandidateTable(plan, candidatePlan, srtSources[0], inputs, out var candidateTable)) { - return false; + return _lastFailureDetail is not null ? false : Detail("buffer candidate table could not be materialized"); } snapshot.BufferCandidateTables.Add(candidateTable); @@ -283,7 +322,7 @@ public static class ResourceMaterializer result = new BufferCandidateTable { MemoryIndex = table.MemoryIndices.Count != 0 ? table.MemoryIndices[0] : -1 }; if (srt.DwordCount != 4) { - return false; + return Detail($"buffer candidate table SRT has {srt.DwordCount} dwords, expected 4: {Words(srt.Dwords)}"); } int count; @@ -295,20 +334,25 @@ public static class ResourceMaterializer { if (table.Limit is null) { - return false; + return Detail("buffer candidate table is not statically bounded and has no limit expression"); } using var scratch = RuntimeEvaluationScratch.Rent(); var evaluator = new RuntimeValueEvaluator(scratch, plan, inputs); - if (!evaluator.Evaluate(table.Limit, out var limit) || !table.TryResolveCount(limit, out count)) + if (!evaluator.Evaluate(table.Limit, out var limit)) { - return false; + return Detail($"buffer candidate table limit cannot be evaluated: {DescribeEvaluationValue(plan, inputs, evaluator, table.Limit)}"); + } + + if (!table.TryResolveCount(limit, out count)) + { + return Detail($"buffer candidate table limit {limit} does not resolve to a count (cap={table.Cap})"); } } if (count <= 0 || count > table.Cap) { - return false; + return Detail($"buffer candidate table count {count} is outside 1..{table.Cap}"); } var keyToCandidate = new Dictionary(); @@ -320,7 +364,7 @@ public static class ResourceMaterializer { if (!ReadScalarBufferWord(srt.Dwords, offset, dword * sizeof(uint), inputs, out words[dword])) { - return false; + return Detail($"buffer candidate {index} dword {dword} (offset {Hex(offset)}): " + DescribeScalarBufferRead(srt.Dwords, offset, dword * sizeof(uint), inputs)); } } @@ -337,7 +381,8 @@ public static class ResourceMaterializer // run time; reject precisely instead of guessing. if (!result.Descriptors[(int)existing].SameAs(descriptor)) { - return false; + return Detail($"buffer candidates {existing} and {index} share probe key 0x{words[0]:X8} but differ: " + + $"{Words(result.Descriptors[(int)existing].Dwords)} vs {Words(words)}"); } continue; @@ -486,6 +531,21 @@ public static class ResourceMaterializer } } + private static readonly HashSet<(uint Pc, uint Stride)> ReportedRecordAlignedFallbacks = new(); + + // Once per image and material stride, so a table probed every draw does not flood the log. + private static void ReportRecordAlignedFallback(uint pc, uint stride, ulong exhaustiveCount, ulong alignedCount) + { + lock (ReportedRecordAlignedFallbacks) + { + if (ReportedRecordAlignedFallbacks.Add((pc, stride))) + { + Console.Error.WriteLine($"[GPU][WARN] indirect image {Hex(pc)}: {exhaustiveCount} unbounded selector probes exceed {MaxIndirectImageProbes}; " + + $"probing the {alignedCount} record-aligned offsets (selectors assumed below 2^32/stride)"); + } + } + } + private static bool NullImageDescriptor(ReadOnlySpan descriptor) => descriptor[0] == 0 && (descriptor[1] & 0xFF) == 0; @@ -493,7 +553,8 @@ public static class ResourceMaterializer { ReadOnlySpan reserved = [ - 0x00000000u, 0x20000000u, 0xf0003000u, 0x00000000u, + // GFX10/10.3 word 2 bit 31 is RESOURCE_LEVEL, not reserved. + 0x00000000u, 0x20000000u, 0x70003000u, 0x00000000u, 0xe000e000u, 0xf9000000u, 0x00007b00u, 0x00000000u, ]; for (var dword = 0; dword < reserved.Length; dword++) @@ -532,6 +593,11 @@ public static class ResourceMaterializer } + if (!BaseAlignedToSwizzleBlock(descriptor)) + { + return false; + } + if (type is GuestImageFormat.ImageType2DMsaa or GuestImageFormat.ImageType2DMsaaArray) { var baseLevel = (descriptor[3] >> 12) & 0xF; @@ -543,6 +609,22 @@ public static class ResourceMaterializer return true; } + // A standard (non-XOR) swizzled surface starts on its swizzle block, so a base inside a + // block cannot describe a real surface. XOR modes fold the pipe/bank XOR into the low + // base bits and are not checked. + private static bool BaseAlignedToSwizzleBlock(ReadOnlySpan descriptor) + { + var swizzleMode = (descriptor[3] >> 20) & 0x1F; + var blockBytes = swizzleMode switch + { + >= 4 and <= 7 => 4UL * 1024, + >= 8 and <= 15 => 64UL * 1024, + _ => 0UL, + }; + var baseAddress = ((ulong)descriptor[0] | ((ulong)(descriptor[1] & 0xFF) << 32)) << 8; + return blockBytes == 0 || (baseAddress & (blockBytes - 1)) == 0; + } + private static bool UsableImageCandidate(ReadOnlySpan candidate, bool r128) => !NullImageDescriptor(candidate) && ValidImageDescriptor(candidate, r128) && ReservedImageBitsClear(candidate) && GuestImageFormat.SampledNumericClass(GuestImageFormat.FormatOf(candidate)) != ImageNumericClass.Unsupported; @@ -573,6 +655,16 @@ public static class ResourceMaterializer return inputs.ReadCleanMemory is not null && inputs.ReadCleanMemory(baseAddress + aligned, out word); } + // Describes a failed ReadScalarBufferWord: the V# words and the address it resolved to. + private static string DescribeScalarBufferRead(ReadOnlySpan descriptor, uint dynamicOffset, uint immediateOffset, ResourceRuntimeInputs inputs) + { + var aligned = ((ulong)dynamicOffset + immediateOffset) & ~3ul; + var baseAddress = ((descriptor[0] | ((ulong)descriptor[1] << 32)) & AddressMask) & ~3ul; + return $"cannot read {sizeof(uint)} bytes at guest address {Hex(baseAddress + aligned)} " + + $"(V# words {Words(descriptor)} base={Hex(baseAddress)} offset={Hex(aligned)} size={Hex(ScalarBufferSize(descriptor))}" + + $"{(inputs.ReadCleanMemory is null ? " no clean reader" : string.Empty)})"; + } + // Enumerates every material key that can pass the table's bounds and reads the // heap descriptor each selects; stale or invalid descriptors become null. private static bool MaterializeIndirectImage( @@ -590,13 +682,13 @@ public static class ResourceMaterializer result = new IndirectImageTable(); if (material.DwordCount != 4 || heap.DwordCount != 4) { - return false; + return Detail($"indirect image {Hex(image.FirstUsePc)}: material/heap descriptors have {material.DwordCount}/{heap.DwordCount} dwords, expected 4/4"); } var materialStride = (material.Dwords[1] >> 16) & 0x3FFF; if (materialStride != indirect.SelectorStride) { - return false; + return Detail($"indirect image {Hex(image.FirstUsePc)}: material V# stride {materialStride} differs from selector stride {indirect.SelectorStride}: {Words(material.Dwords)}"); } var period = 1ul << 32; @@ -619,20 +711,45 @@ public static class ResourceMaterializer diagnostic.ProvenOffsets = provenOffsets ?? []; } } + // Without a proven selector set every offset residue + k * step is reachable, because + // selector * stride + offset wraps modulo 2^32 for an unbounded 32-bit selector. A + // table whose exhaustive probe count fits the cap keeps that enumeration unchanged. + // Past the cap, a non-multiple of the stride needs a selector >= 2^32 / stride, which + // no real table index reaches (the hardware reads selector * stride + offset), so the + // probe falls back to the record-aligned offsets offset + k * stride inside the table. + var probeStart = residue; + var probeStep = step; + var probeTotal = probeCount; if (provenOffsets is null && probeCount > MaxIndirectImageProbes) { - return false; + var alignedCount = indirect.SelectorStride != 0 && indirect.SelectorOffset <= limit ? + (limit - indirect.SelectorOffset) / indirect.SelectorStride + 1 : 0; + if (indirect.SelectorStride == 0 || alignedCount > MaxIndirectImageProbes) + { + return Detail($"indirect image {Hex(image.FirstUsePc)}: {probeCount} unbounded selector probes exceed {MaxIndirectImageProbes} (material V# {Words(material.Dwords)} stride={indirect.SelectorStride} offset={indirect.SelectorOffset})"); + } + + ReportRecordAlignedFallback(image.FirstUsePc, indirect.SelectorStride, probeCount, alignedCount); + if (diagnostic is not null) + { + diagnostic.SelectionMode = "record_aligned_fallback"; + } + + probeStart = indirect.SelectorOffset; + probeStep = indirect.SelectorStride; + probeTotal = alignedCount; } var keys = new List { 0 }; var seen = new HashSet { 0 }; var offsets = provenOffsets is not null ? provenOffsets.Select(offset => (ulong)offset) : - Enumerable.Range(0, (int)probeCount).Select(index => residue + (ulong)index * step); + Enumerable.Range(0, (int)probeTotal).Select(index => probeStart + (ulong)index * probeStep); foreach (var offset in offsets) { if (!ReadScalarBufferWord(material.Dwords, (uint)offset, indirect.MaterialImmediate, inputs, out var key)) { - return false; + return Detail($"indirect image {Hex(image.FirstUsePc)} material key at offset {Hex(offset)}: " + + DescribeScalarBufferRead(material.Dwords, (uint)offset, indirect.MaterialImmediate, inputs)); } if (seen.Add(key)) @@ -648,11 +765,13 @@ public static class ResourceMaterializer { var candidate = new uint[8]; var heapOffset = key << 5; - for (uint dword = 0; dword < 8; dword++) + if (!ScalarBufferRangeRead.TryRead(heap.Dwords, heapOffset, 0, inputs, candidate)) { - if (!ReadScalarBufferWord(heap.Dwords, heapOffset, dword * sizeof(uint), inputs, out candidate[dword])) + for (uint dword = 0; dword < 8; dword++) { - return false; + if (!ReadScalarBufferWord(heap.Dwords, heapOffset, dword * sizeof(uint), inputs, out candidate[dword])) + return Detail($"indirect image {Hex(image.FirstUsePc)} heap descriptor for key 0x{key:X8} dword {dword}: " + + DescribeScalarBufferRead(heap.Dwords, heapOffset, dword * sizeof(uint), inputs)); } } @@ -692,8 +811,12 @@ public static class ResourceMaterializer { failure = ResourceMaterializationFailure.Other; result = new IndirectImageTable(); + if (indirect.BufferTableStride != 0) + return MaterializeBufferTableImage(indirect, heap, r128, inputs, out result, out failure); + if (heap.DwordCount != 2 || indirect.KeyBound == 0 || indirect.KeyBound > MaxIndirectImageProbes || inputs.ReadCleanMemory is null) - return false; + return Detail($"dense indirect image {Hex(image.FirstUsePc)}: heap dwords={heap.DwordCount} (expected 2) keyBound={indirect.KeyBound} " + + $"(1..{MaxIndirectImageProbes}) cleanReader={inputs.ReadCleanMemory is not null}"); var baseAddress = (((ulong)heap.Dwords[1] << 32) | heap.Dwords[0]) & AddressMask; var probed = new List((int)indirect.KeyBound); @@ -706,7 +829,8 @@ public static class ResourceMaterializer var relative = entry + dword * sizeof(uint); if (relative > AddressMask || baseAddress > AddressMask - relative || !inputs.ReadCleanMemory(baseAddress + relative, out candidate[dword])) - return false; + return Detail($"dense indirect image {Hex(image.FirstUsePc)} key {key} dword {dword}: cannot read {sizeof(uint)} bytes at guest address " + + $"{Hex(baseAddress + relative)} (heap base={Hex(baseAddress)} offset={Hex(relative)})"); } if (!UsableImageCandidate(candidate, r128)) @@ -722,6 +846,129 @@ public static class ResourceMaterializer out failure); } + // Every record of a V# table: the key is the record's byte offset, which the shader + // passes as its scalar buffer offset. The V# range bounds the records; a record whose + // bytes cannot be read ends the table as the hardware would read out of range. + private static bool MaterializeBufferTableImage( + IndirectImageSelector indirect, + DescriptorWords table, + bool r128, + ResourceRuntimeInputs inputs, + out IndirectImageTable result, + out ResourceMaterializationFailure failure) + { + failure = ResourceMaterializationFailure.Other; + result = new IndirectImageTable(); + if (table.DwordCount != 4 || inputs.ReadCleanMemory is null) + return Detail($"buffer-table image: table has {table.DwordCount} dwords (expected 4), cleanReader={inputs.ReadCleanMemory is not null}"); + + var baseAddress = (table.Dwords[0] | ((ulong)(table.Dwords[1] & 0xFFFF) << 32)) & AddressMask; + var recordStride = (table.Dwords[1] >> 16) & 0x3FFF; + var rangeBytes = recordStride == 0 ? (ulong)table.Dwords[2] : (ulong)table.Dwords[2] * recordStride; + // A pointer record holds a 64-bit address; a direct record holds the descriptor. + var pointerTable = indirect.PointerTargetOffset is not null; + var entryBytes = pointerTable ? (ulong)sizeof(ulong) : 8 * sizeof(uint); + // Out-of-range buffer reads return zero on hardware: an empty table selects null. + var count = rangeBytes < indirect.TableOffset + entryBytes ? 0 : Math.Min((rangeBytes - indirect.TableOffset - entryBytes) / indirect.BufferTableStride + 1, MaxIndirectImageProbes); + var probed = new List(); + var keys = new List(); + for (ulong record = 0; record < count; record++) + { + var candidate = new uint[8]; + var entry = indirect.TableOffset + record * indirect.BufferTableStride; + var readable = true; + var dwords = r128 ? 4u : 8u; + if (pointerTable) + { + if (!TryReadCleanWord(baseAddress, entry, inputs, out var low) || + !TryReadCleanWord(baseAddress, entry + sizeof(uint), inputs, out var high)) + break; + + // A null or unmapped pointer cannot be followed; that record selects null. + var target = (((ulong)high << 32) | low) & AddressMask; + for (uint dword = 0; dword < dwords && readable && target != 0; dword++) + readable = TryReadCleanWord(target, indirect.PointerTargetOffset!.Value + dword * sizeof(uint), inputs, out candidate[dword]); + if (target == 0 || !readable) + Array.Clear(candidate); + } + else + { + for (uint dword = 0; dword < dwords && readable; dword++) + readable = TryReadCleanWord(baseAddress, entry + dword * sizeof(uint), inputs, out candidate[dword]); + if (!readable) + break; + } + + if (NullImageDescriptor(candidate) || !ValidImageDescriptor(candidate, r128) || !ReservedImageBitsClear(candidate) || + !ImageBaseMapped(candidate, inputs)) + Array.Clear(candidate); + probed.Add(candidate); + keys.Add(checked((uint)(record * indirect.BufferTableStride))); + } + + if (probed.Count == 0) + { + probed.Add(new uint[8]); + keys.Add(0); + } + + return FinishIndirectImage(probed, keys, out result, out failure); + } + + // One sampler for a pointer table: every record whose pointer can be followed must hold + // the same four dwords, or no single host sampler represents what the shader selects. + private static bool MaterializePointerTableSampler( + PointerTableSelector selector, + DescriptorWords table, + ResourceRuntimeInputs inputs, + out uint[] sampler) + { + sampler = new uint[4]; + if (table.DwordCount != 4 || inputs.ReadCleanMemory is null || selector.Stride == 0) + return Detail($"pointer-table sampler: table has {table.DwordCount} dwords (expected 4), cleanReader={inputs.ReadCleanMemory is not null}, stride={selector.Stride}"); + + var baseAddress = (table.Dwords[0] | ((ulong)(table.Dwords[1] & 0xFFFF) << 32)) & AddressMask; + var recordStride = (table.Dwords[1] >> 16) & 0x3FFF; + var rangeBytes = recordStride == 0 ? (ulong)table.Dwords[2] : (ulong)table.Dwords[2] * recordStride; + if (rangeBytes < selector.TableOffset + sizeof(ulong)) + return true; + + var count = Math.Min((rangeBytes - selector.TableOffset - sizeof(ulong)) / selector.Stride + 1, MaxIndirectImageProbes); + uint[]? common = null; + var candidate = new uint[4]; + for (ulong record = 0; record < count; record++) + { + var entry = selector.TableOffset + record * selector.Stride; + if (!TryReadCleanWord(baseAddress, entry, inputs, out var low) || + !TryReadCleanWord(baseAddress, entry + sizeof(uint), inputs, out var high)) + break; + + var target = (((ulong)high << 32) | low) & AddressMask; + var readable = target != 0; + for (uint dword = 0; dword < 4 && readable; dword++) + readable = TryReadCleanWord(target, selector.TargetOffset + dword * sizeof(uint), inputs, out candidate[dword]); + if (!readable) + continue; + + if (common is null) + common = (uint[])candidate.Clone(); + else if (!common.AsSpan().SequenceEqual(candidate)) + return Fail($"pointer-table sampler differs between records: record={record} stride={selector.Stride} offset={selector.TargetOffset}"); + } + + if (common is not null) + sampler = common; + return true; + } + + // A bindless table names every texture the game knows; one whose memory is not mapped is + // not resident, and sampling it cannot be valid, so it binds as null. + private static bool ImageBaseMapped(ReadOnlySpan descriptor, ResourceRuntimeInputs inputs) + { + var baseAddress = (((ulong)descriptor[0] | ((ulong)(descriptor[1] & 0xFF) << 32)) << 8) & AddressMask; + return baseAddress == 0 || inputs.ReadMemory is null || inputs.ReadMemory(baseAddress, out _); + } + private static bool MaterializeWaveIndexedImage( IndirectImageSelector indirect, WaveIndexedImageSelector wave, @@ -735,11 +982,12 @@ public static class ResourceMaterializer failure = ResourceMaterializationFailure.Other; result = new IndirectImageTable(); if (heap.DwordCount != 2 || inputs.ReadCleanMemory is null || wave.IndexStride == 0) - return false; + return Detail($"wave-indexed image {Hex(image.FirstUsePc)}: heap dwords={heap.DwordCount} (expected 2) " + + $"cleanReader={inputs.ReadCleanMemory is not null} indexStride={wave.IndexStride}"); var baseAddress = (((ulong)heap.Dwords[1] << 32) | heap.Dwords[0]) & AddressMask; if (!TryReadCleanWord(baseAddress, wave.MaskOffset, inputs, out var activeMask)) - return false; + return ReadFailed($"wave-indexed image {Hex(image.FirstUsePc)} active mask", baseAddress, wave.MaskOffset, inputs); var keys = new SortedSet(); for (uint bit = 0; bit < 32; bit++) @@ -748,7 +996,7 @@ public static class ResourceMaterializer continue; var indexOffset = (ulong)wave.IndexTableOffset + (ulong)bit * wave.IndexStride; if (!TryReadCleanWord(baseAddress, indexOffset, inputs, out var key)) - return false; + return ReadFailed($"wave-indexed image {Hex(image.FirstUsePc)} lane {bit} index", baseAddress, indexOffset, inputs); // VCmpxLeI32 0, key suppresses signed-negative indices before ReadFirstLane. if ((key & 0x8000_0000u) == 0) keys.Add(key); @@ -760,10 +1008,14 @@ public static class ResourceMaterializer { var candidate = new uint[8]; var descriptorOffset = (ulong)indirect.TableOffset + ((ulong)key << 5); - for (uint dword = 0; dword < candidate.Length; dword++) + if (!ScalarBufferRangeRead.TryReadAddress(baseAddress, descriptorOffset, inputs, candidate)) { - if (!TryReadCleanWord(baseAddress, descriptorOffset + dword * sizeof(uint), inputs, out candidate[dword])) - return false; + for (uint dword = 0; dword < candidate.Length; dword++) + { + if (!TryReadCleanWord(baseAddress, descriptorOffset + dword * sizeof(uint), inputs, out candidate[dword])) + return ReadFailed($"wave-indexed image {Hex(image.FirstUsePc)} key {key} descriptor dword {dword}", + baseAddress, descriptorOffset + dword * sizeof(uint), inputs); + } } if (!UsableImageCandidate(candidate, r128)) @@ -797,19 +1049,21 @@ public static class ResourceMaterializer failure = ResourceMaterializationFailure.Other; result = new IndirectImageTable(); result.Keys.AddRange(keys); + // A bindless table holds tens of thousands of candidates; index them by content. + var indices = new Dictionary(DescriptorContentComparer.Instance); foreach (var candidate in probed) { var words = new DescriptorWords(candidate); - var found = result.Descriptors.FindIndex(existing => existing.SameAs(words)); - if (found < 0) + if (!indices.TryGetValue(candidate, out var found)) { if (result.Descriptors.Count >= ShaderResourceInfo.MaxImages) { failure = ResourceMaterializationFailure.ImageCapacityExceeded; - return false; + return Fail($"indirect image table has more than {ShaderResourceInfo.MaxImages} distinct descriptors among {probed.Count} candidates"); } found = result.Descriptors.Count; result.Descriptors.Add(words); + indices.Add(candidate, found); } result.Candidates.Add((uint)found); } @@ -868,12 +1122,65 @@ public static class ResourceMaterializer return baseLevel <= last ? last - baseLevel + 1 : 0; } + // Whether an instruction declaring one dimension addresses a descriptor of another: + // arrayness can differ, the spatial dimensionality and sample layout cannot. + private static bool DimensionAddressable(ImageDimension descriptor, ImageDimension declared) => + descriptor == declared || (Family(descriptor) != 0 && Family(descriptor) == Family(declared)); + + private static int Family(ImageDimension dimension) => dimension switch + { + ImageDimension.Dim1D or ImageDimension.Dim1DArray => 1, + ImageDimension.Dim2D or ImageDimension.Dim2DArray => 2, + ImageDimension.Dim3D => 3, + ImageDimension.Dim2DMsaa or ImageDimension.Dim2DMsaaArray => 4, + _ => 0, + }; + private static bool Fail(string message) { + _lastFailureDetail = message; SpecializationFailed(message); return false; } + // Records why the current Materialize call failed and returns false. Callers build the + // text only on their failure path. + private static bool Detail(string message) + { + _lastFailureDetail = message; + return false; + } + + private static string Hex(ulong value) => $"0x{value:X}"; + + private static string Words(ReadOnlySpan words) => string.Join(",", words.ToArray().Select(word => $"{word:x8}")); + + // The first dword of the given sources that cannot be evaluated, with the values behind it. + private static string DiagnoseSources(ShaderResourcePlan plan, ResourceRuntimeInputs inputs, IReadOnlyList sources) + { + var evaluator = new RuntimeValueEvaluator(plan, inputs); + foreach (var sourceIndex in sources) + { + if (sourceIndex >= plan.DescriptorSources.Count) + return $"descriptor source {sourceIndex} is outside the source table"; + var source = plan.DescriptorSources[(int)sourceIndex]; + for (var dword = 0; dword < source.Dwords.Length; dword++) + { + if (!evaluator.Evaluate(source.Dwords[dword], out _)) + return $"descriptor source {sourceIndex} dword {dword} cannot be evaluated: " + + DescribeEvaluationValue(plan, inputs, evaluator, source.Dwords[dword]); + } + } + + return $"descriptor sources [{string.Join(",", sources)}] failed without an isolated dword"; + } + + // Reports a guest word that could not be read: the address and size, split into the + // base and offset the caller used. + private static bool ReadFailed(string step, ulong baseAddress, ulong offset, ResourceRuntimeInputs inputs) => + Detail($"{step}: cannot read {sizeof(uint)} bytes at guest address {Hex(baseAddress + offset)} " + + $"(base={Hex(baseAddress)} offset={Hex(offset)}{(inputs.ReadCleanMemory is null ? " no clean reader" : string.Empty)})"); + private static bool BuildSpecialization( ShaderResourcePlan plan, MaterializedSnapshot snapshot, @@ -899,7 +1206,7 @@ public static class ResourceMaterializer if (imageCount + table.Descriptors.Count - 1 > ShaderResourceInfo.MaxImages) { failure = ResourceMaterializationFailure.ImageCapacityExceeded; - return Fail("indirect image candidates exceed the dense image resource limit"); + return Fail($"indirect image candidates exceed the dense image resource limit: images={imageCount} candidates={table.Descriptors.Count} keys={table.Keys.Count} limit={ShaderResourceInfo.MaxImages}"); } imageCount += table.Descriptors.Count - 1; @@ -1056,7 +1363,7 @@ public static class ResourceMaterializer var numericClass = GuestImageFormat.SampledNumericClass(format); if (storage) { - if ((!rawSintStorage && numericClass == ImageNumericClass.Sint) || numericClass == ImageNumericClass.Unsupported) + if (numericClass == ImageNumericClass.Unsupported) { return Fail($"storage image descriptor {index} uses unsupported format {format}"); } @@ -1112,6 +1419,10 @@ public static class ResourceMaterializer return Fail("indirect image specialization has an invalid key mapping"); } + // The instruction's MIMG dimension bounds what a candidate can be: a descriptor of + // another dimension is never a valid selection for it, so it binds as null. + var declaredDimension = info.Images[rootIndex].Dimension; + var declaredCube = info.Images[rootIndex].Cube; var exemplar = DescriptorConstants.NoIndex; var resourceCount = 0; for (var resource = 0; resource < images.Count; resource++) @@ -1122,18 +1433,35 @@ public static class ResourceMaterializer } resourceCount++; - if (exemplar == DescriptorConstants.NoIndex && !NullImageDescriptor(snapshot.Images[resource])) + if (exemplar == DescriptorConstants.NoIndex && !NullImageDescriptor(snapshot.Images[resource]) && + DimensionAddressable(images[resource].Dimension, declaredDimension) && images[resource].Cube == declaredCube) { exemplar = (uint)resource; } } - if (resourceCount < 2 || exemplar == DescriptorConstants.NoIndex) + if (resourceCount < 2) { - return Fail("indirect image specialization has no typed candidate"); + return Fail("indirect image specialization has fewer than two candidates"); } - var imageClass = images[(int)exemplar]; + var declaredImage = info.Images[rootIndex]; + // A bounded table can contain only descriptors from another image family. Keep + // the instruction's declared type and bind every candidate as a typed null. + var imageClass = exemplar == DescriptorConstants.NoIndex + ? images[rootIndex] with + { + NumericClass = declaredImage.NumericClass == ImageNumericClass.Unsupported + ? declaredImage.Atomic ? ImageNumericClass.Uint : ImageNumericClass.Float + : declaredImage.NumericClass, + Dimension = declaredDimension, + MipCount = declaredImage.MipCount, + ConversionFormat = declaredImage.ConversionFormat, + ShaderSwizzle = declaredImage.ShaderSwizzle, + Cube = declaredCube, + EmulatedCompareFunction = declaredImage.EmulatedCompareFunction, + } + : images[(int)exemplar]; var separateSampledDimensions = info.Images[rootIndex].ResourceClass == ImageResourceClass.Sampled; for (var candidate = 0; candidate < images.Count; candidate++) { @@ -1162,7 +1490,24 @@ public static class ResourceMaterializer separateSampledDimensions && !image.Cube && !imageClass.Cube && image.Dimension is ImageDimension.Dim2D or ImageDimension.Dim2DArray && imageClass.Dimension is ImageDimension.Dim2D or ImageDimension.Dim2DArray; - if (image.NumericClass != imageClass.NumericClass || !compatibleDimensions || + if (!DimensionAddressable(image.Dimension, declaredDimension) || image.Cube != declaredCube) + { + snapshot.Images[candidate] = new uint[snapshot.Images[candidate].Length]; + images[candidate] = image with + { + NumericClass = imageClass.NumericClass, + Dimension = imageClass.Dimension, + MipCount = imageClass.MipCount, + ConversionFormat = imageClass.ConversionFormat, + ShaderSwizzle = imageClass.ShaderSwizzle, + Cube = imageClass.Cube, + }; + continue; + } + + // Each candidate is emitted as its own case with its own typed binding, so the + // numeric class may differ between candidates, as the hardware allows. + if (!compatibleDimensions || image.MipCount != imageClass.MipCount || image.ConversionFormat != imageClass.ConversionFormat || image.ShaderSwizzle != imageClass.ShaderSwizzle || image.Cube != imageClass.Cube) { @@ -1564,3 +1909,43 @@ public static class ResourceMaterializer return new SpecializedResourceInfo { Info = info, SamplerByMemoryIndex = samplerByMemory }; } } + +internal static class ScalarBufferRangeRead +{ + internal static bool TryReadAddress(ulong baseAddress, ulong offset, + ResourceRuntimeInputs inputs, Span words) + { + const ulong addressMask = 0x0000_FFFF_FFFF_FFFFul; + var length = (ulong)words.Length * sizeof(uint); + return inputs.ReadCleanWords is not null && length != 0 && baseAddress <= addressMask && + offset <= addressMask - baseAddress && length - 1 <= addressMask - baseAddress - offset && + inputs.ReadCleanWords(baseAddress + offset, words); + } + + // Use one clean read only when every scalar load is in bounds and contiguous. + // A refusal leaves the caller's original word reads and OOB handling intact. + internal static bool TryRead(ReadOnlySpan descriptor, uint dynamicOffset, + uint immediateOffset, ResourceRuntimeInputs inputs, Span words) + { + if (inputs.ReadCleanWords is null || descriptor.Length != 4 || words.IsEmpty) + return false; + + var lastImmediate = (ulong)immediateOffset + (ulong)(words.Length - 1) * sizeof(uint); + if (lastImmediate > uint.MaxValue) + return false; + + const ulong addressMask = 0x0000_FFFF_FFFF_FFFFul; + var stride = (descriptor[1] >> 16) & 0x3FFF; + var size = stride == 0 ? descriptor[2] : (ulong)stride * descriptor[2]; + var offset = ((ulong)dynamicOffset + immediateOffset) & ~3ul; + var length = (ulong)words.Length * sizeof(uint); + if (offset > size || length > size - offset) + return false; + + var baseAddress = ((descriptor[0] | ((ulong)descriptor[1] << 32)) & addressMask) & ~3ul; + if (offset > addressMask - baseAddress || length - 1 > addressMask - baseAddress - offset) + return false; + + return inputs.ReadCleanWords(baseAddress + offset, words); + } +} diff --git a/src/SharpEmu.ShaderCompiler/Resources/ResourceRuntimeInputs.cs b/src/SharpEmu.ShaderCompiler/Resources/ResourceRuntimeInputs.cs index a6ee7e1..8eb22e8 100644 --- a/src/SharpEmu.ShaderCompiler/Resources/ResourceRuntimeInputs.cs +++ b/src/SharpEmu.ShaderCompiler/Resources/ResourceRuntimeInputs.cs @@ -7,15 +7,18 @@ namespace SharpEmu.ShaderCompiler.Resources; // Reads one dword of guest memory; false when the address cannot be read. public delegate bool GuestWordReader(ulong address, out uint word); +// A contiguous clean range; false makes the caller use its individual word reads. +public delegate bool GuestWordsReader(ulong address, Span words); // What one draw supplies to materialise a plan: its user data, the shader base and // the two memory readers. The clean reader refuses memory the GPU may still own. -public sealed class ResourceRuntimeInputs +public sealed record ResourceRuntimeInputs { public IReadOnlyList UserData { get; init; } = []; public ulong ShaderBase { get; init; } public GuestWordReader? ReadMemory { get; init; } public GuestWordReader? ReadCleanMemory { get; init; } + public GuestWordsReader? ReadCleanWords { get; init; } public ComputeSelectorState? ComputeState { get; init; } // Told true before the flattened table's words are evaluated and false after, so a reader @@ -26,10 +29,8 @@ public sealed class ResourceRuntimeInputs public bool ReadsClean { get; init; } - public ResourceRuntimeInputs WithReader(GuestWordReader? reader) => new() + public ResourceRuntimeInputs WithReader(GuestWordReader? reader) => this with { - UserData = UserData, - ShaderBase = ShaderBase, ReadMemory = reader, ReadCleanMemory = ReadCleanMemory, ComputeState = ComputeState, diff --git a/src/SharpEmu.ShaderCompiler/Resources/ResourceSpecialization.cs b/src/SharpEmu.ShaderCompiler/Resources/ResourceSpecialization.cs index f1fbcd9..7bb0ce6 100644 --- a/src/SharpEmu.ShaderCompiler/Resources/ResourceSpecialization.cs +++ b/src/SharpEmu.ShaderCompiler/Resources/ResourceSpecialization.cs @@ -6,7 +6,16 @@ using System.Linq; namespace SharpEmu.ShaderCompiler.Resources; -public readonly record struct BufferSpecialization(uint PackedStride, uint DescriptorFormat, uint DescriptorSwizzle); +public readonly record struct BufferSpecialization(uint PackedStride, uint DescriptorFormat, uint DescriptorSwizzle) +{ + public const uint SwizzleEnabledBit = 1u << 14; + public const uint StrideMask = 0x3FFF; + + // With runtime strides only a swizzled buffer keeps its stride compiled in: swizzled + // addressing derives its element layout from it. The flag bits always stay. + public BufferSpecialization WithoutRuntimeStride() => + (PackedStride & SwizzleEnabledBit) != 0 ? this : this with { PackedStride = PackedStride & ~StrideMask }; +} // The module-affecting layout of one bounded runtime V# table. Candidate descriptors // themselves are appended to Buffers; this names their contiguous run and the flattened diff --git a/src/SharpEmu.ShaderCompiler/Resources/ResourceTableReadPlanner.cs b/src/SharpEmu.ShaderCompiler/Resources/ResourceTableReadPlanner.cs index 60d73a9..c7884cd 100644 --- a/src/SharpEmu.ShaderCompiler/Resources/ResourceTableReadPlanner.cs +++ b/src/SharpEmu.ShaderCompiler/Resources/ResourceTableReadPlanner.cs @@ -148,6 +148,14 @@ public sealed class ResourceTableReadPlanner return; } + // An address carried by a loop phi that changes per iteration is the loop's current + // pointer: the host sees one value per draw, the shader one per iteration, so the + // shader performs the read. A phi that only merges one value with itself is that value. + if (DependsOnVaryingPhi(value.Operands[0])) + { + return; + } + for (var slot = 0; slot < _reads.Count; slot++) { if (_graph.Equivalent(value, _reads[slot].Value)) @@ -162,6 +170,30 @@ public sealed class ResourceTableReadPlanner _patches.Add((value, newSlot)); } + private bool DependsOnVaryingPhi(ScalarValue value) + { + var pending = new Stack(); + var visited = new HashSet(); + pending.Push(value); + while (pending.TryPop(out var current)) + { + if (!visited.Add(current)) + continue; + if (current.Kind == ScalarValueKind.Phi) + { + if (_graph.ResolveInvariantPhi(current) is not { } invariant || invariant.Kind == ScalarValueKind.Phi) + return true; + pending.Push(invariant); + continue; + } + + foreach (var operand in current.Operands) + pending.Push(operand); + } + + return false; + } + // Equivalent reads share one host read, but each instruction must load its destination // from that slot instead of reading guest memory again on the device. private void PatchReads() diff --git a/src/SharpEmu.ShaderCompiler/Resources/ResourceTracker.DenseImages.cs b/src/SharpEmu.ShaderCompiler/Resources/ResourceTracker.DenseImages.cs index 284d3e5..d993ac7 100644 --- a/src/SharpEmu.ShaderCompiler/Resources/ResourceTracker.DenseImages.cs +++ b/src/SharpEmu.ShaderCompiler/Resources/ResourceTracker.DenseImages.cs @@ -25,7 +25,9 @@ public sealed partial class ResourceTracker for (var dword = 0; dword < 8; dword++) { - var read = handle.Operands[dword]; + // A descriptor kept in its registers across a loop reaches the handle through + // phis that only carry the loaded value. + var read = _graph.ResolveInvariantPhi(handle.Operands[dword]) ?? handle.Operands[dword]; if (read.Kind != ScalarValueKind.ScalarAddressWord || read.MemoryIndex < 0 || read.MemoryIndex >= _plan.Memory.Count || !MemoryIndexBelongsTo(read.MemoryIndex, read)) return false; @@ -34,7 +36,7 @@ public sealed partial class ResourceTracker if (memory.Kind != MemoryResourceKind.ScalarAddress || memory.DataBits != 32 || memory.DataDwords != 1) return false; - var offset = read.Operands[1]; + var offset = _graph.ResolveInvariantPhi(read.Operands[1]) ?? read.Operands[1]; uint extra = 0; if (based is null) { @@ -113,7 +115,8 @@ public sealed partial class ResourceTracker scaled.Operands[1].ConstantU32 != DenseIndirectImageShift) return false; - var key = scaled.Operands[0]; + // A key carried unchanged around a loop is bounded by the value it carries. + var key = _graph.ResolveInvariantPhi(scaled.Operands[0]) ?? scaled.Operands[0]; var bound = BoundBySamplerLoads(DenseKeyBound(key), heapHandle, tableOffset); var waveIndexed = TryCreateWaveIndexedImageSelector(key, reads); if (bound == 0 && waveIndexed is null) @@ -121,8 +124,10 @@ public sealed partial class ResourceTracker return false; } + // Several image instructions may sample the same loaded descriptor; each builds its + // own handle over these reads and gets the same plan. foreach (var read in reads) - if (!UsesOnly(read, [handle])) + if (!UsedOnlyByImageHandlesOver(read, reads)) return false; if (!MakeRuntimeAddressSource(heapHandle, pc, out var heapSourceIndex, out var heapSource)) @@ -158,6 +163,39 @@ public sealed partial class ResourceTracker return true; } + private bool UsedOnlyByImageHandlesOver(ScalarValue read, IReadOnlyList reads) + { + var pending = new Stack(); + var visited = new HashSet(); + pending.Push(read); + while (pending.TryPop(out var value)) + { + if (!visited.Add(value) || !_uses.TryGetValue(value, out var uses) || uses.Count == 0) + continue; + + foreach (var user in uses) + { + if (user.Kind == ScalarValueKind.Phi && ReferenceEquals(_graph.ResolveInvariantPhi(user), read)) + { + pending.Push(user); + continue; + } + + if (user.Kind != ScalarValueKind.ImageHandle || user.Operands.Length != reads.Count) + return false; + + for (var index = 0; index < reads.Count; index++) + { + var operand = _graph.ResolveInvariantPhi(user.Operands[index]) ?? user.Operands[index]; + if (!ReferenceEquals(operand, reads[index])) + return false; + } + } + } + + return true; + } + private uint BoundBySamplerLoads(uint bound, ScalarValue heapHandle, uint tableOffset) { foreach (var access in _plan.Accesses) @@ -189,6 +227,18 @@ public sealed partial class ResourceTracker _graph.HasNonZeroBitScanInput(key)) return 32; + // A constant-width bitfield, or a value masked to its low bits, has 2^width values. + if (key.Kind == ScalarValueKind.Operation && key.Operation == ScalarOperation.BitFieldUExtract && + key.Operands.Length == 3 && key.Operands[2].IsConstant && key.Operands[2].ConstantU32 is > 0 and <= 12) + return 1u << (int)key.Operands[2].ConstantU32; + + if (key.Kind == ScalarValueKind.Operation && key.Operation == ScalarOperation.And32 && key.Operands.Length == 2) + { + var mask = key.Operands[1].IsConstant ? key.Operands[1] : key.Operands[0].IsConstant ? key.Operands[0] : null; + if (mask is not null && mask.ConstantU32 < MaxDenseIndirectImageEntries && (mask.ConstantU32 & (mask.ConstantU32 + 1)) == 0) + return mask.ConstantU32 + 1; + } + if (!_uses.TryGetValue(key, out var uses)) return 0; @@ -573,4 +623,228 @@ public sealed partial class ResourceTracker constant = 0; return false; } + + // A descriptor loaded with s_buffer_load from a table of fixed-size records, at a + // record offset the shader computes at run time (key * stride). The hardware reads + // whichever record the offset names; the host binds every image the table holds and + // the shader selects one by the same offset. + private bool TryMakeBufferTableImage(ScalarValue handle, uint pc, bool r128, out IndirectImagePlan plan) + { + plan = null!; + if (handle.Kind != ScalarValueKind.ImageHandle || handle.Operands.Length != 8) + return false; + + // A 128-bit resource reads only the first four dwords of the handle. + var dwords = r128 ? 4 : 8; + var reads = new ScalarValue[dwords]; + var memoryIndices = new int[dwords]; + ScalarValue? tableHandle = null; + ScalarValue? offset = null; + uint tableImmediate = 0; + for (var dword = 0; dword < dwords; dword++) + { + var read = handle.Operands[dword]; + var memory = ScalarReadMemory(read, out var memoryIndex); + if (memory is null || memory.Kind != MemoryResourceKind.ScalarBuffer || !MemoryIndexBelongsTo(memoryIndex, read)) + return false; + + var componentOffset = (uint)dword * sizeof(uint); + if (memory.Offset < componentOffset) + return false; + if (dword == 0) + tableImmediate = memory.Offset; + else if (memory.Offset - componentOffset != tableImmediate) + return false; + + if (tableHandle is null) + { + tableHandle = read.Operands[0]; + offset = read.Operands[1]; + } + else if (!ReferenceEquals(read.Operands[0], tableHandle) || !_graph.Equivalent(read.Operands[1], offset!)) + { + return false; + } + + reads[dword] = read; + memoryIndices[dword] = memoryIndex; + } + + if (tableHandle is null || offset is null || !TryGetRecordStride(offset, out var stride)) + return false; + + // Other consumers of the same words (a sampler or another image in the record) keep + // the loads; they are ordinary buffer reads at the offset the shader computes. + var suppressReads = reads.All(read => UsesOnly(read, [handle])); + if (!MakeRuntimeBufferSource(tableHandle, pc, out var tableSourceIndex, out var tableSource)) + return false; + + var imageSource = new DescriptorSource + { + Dwords = [.. tableSource.Dwords, .. tableSource.Dwords], + IndirectImage = new IndirectImageSelector(0, tableSourceIndex, 0, 0, 0) + { + Dense = true, + TableOffset = tableImmediate, + BufferTableStride = stride, + }, + }; + + plan = new IndirectImagePlan + { + Handle = handle, + Source = InternSource(imageSource), + Key = reads[0], + KeyIsAddressOffset = true, + HeapSource = tableSourceIndex, + SuppressMemoryReads = suppressReads, + Memory = memoryIndices, + Reads = reads, + }; + return true; + } + + // A descriptor loaded through a pointer that a V# table record holds: the shader reads + // the record's 64-bit pointer with S_BUFFER_LOAD at key * stride, then the descriptor + // with S_LOAD at a constant offset from that pointer. The key is the record offset. + private sealed record PointerTableMatch( + ScalarValue TableHandle, + ScalarValue KeyRead, + uint TableOffset, + uint Stride, + uint TargetOffset, + ScalarValue[] Reads, + int[] MemoryIndices); + + private bool TryMatchPointerTable(ScalarValue handle, int dwords, out PointerTableMatch match) + { + match = null!; + if (handle.Operands.Length < dwords) + return false; + + var reads = new ScalarValue[dwords]; + var memoryIndices = new int[dwords]; + ScalarValue? address = null; + uint targetOffset = 0; + for (var dword = 0; dword < dwords; dword++) + { + var read = handle.Operands[dword]; + if (read.Kind != ScalarValueKind.ScalarAddressWord || read.Operands.Length != 2 || + read.MemoryIndex >= _plan.Memory.Count || !read.Operands[1].IsConstant || read.Operands[1].ConstantU32 != 0) + return false; + + var memory = _plan.Memory[read.MemoryIndex]; + if (memory.Kind != MemoryResourceKind.ScalarAddress || memory.DataBits != 32 || memory.DataDwords != 1 || + !MemoryIndexBelongsTo(read.MemoryIndex, read)) + return false; + + var componentOffset = (uint)dword * sizeof(uint); + if (memory.Offset < componentOffset) + return false; + if (dword == 0) + { + targetOffset = memory.Offset; + address = read.Operands[0]; + } + else if (memory.Offset - componentOffset != targetOffset || !ReferenceEquals(read.Operands[0], address)) + { + return false; + } + + reads[dword] = read; + memoryIndices[dword] = read.MemoryIndex; + } + + if (address is not { Kind: ScalarValueKind.AddressHandle, Operands.Length: 2 }) + return false; + + var low = _graph.ResolveInvariantPhi(address.Operands[0]); + var high = _graph.ResolveInvariantPhi(address.Operands[1]); + if (low is null || high is null) + return false; + + var lowMemory = ScalarReadMemory(low, out _); + var highMemory = ScalarReadMemory(high, out _); + if (lowMemory is null || highMemory is null || lowMemory.Kind != MemoryResourceKind.ScalarBuffer || + highMemory.Kind != MemoryResourceKind.ScalarBuffer || highMemory.Offset != lowMemory.Offset + sizeof(uint) || + !ReferenceEquals(low.Operands[0], high.Operands[0]) || !_graph.Equivalent(low.Operands[1], high.Operands[1]) || + !TryGetRecordStride(low.Operands[1], out var stride)) + return false; + + match = new PointerTableMatch(low.Operands[0], low, lowMemory.Offset, stride, targetOffset, reads, memoryIndices); + return true; + } + + private bool TryMakePointerTableImage(ScalarValue handle, uint pc, bool r128, out IndirectImagePlan plan) + { + plan = null!; + if (handle.Kind != ScalarValueKind.ImageHandle || handle.Operands.Length != 8 || + !TryMatchPointerTable(handle, r128 ? 4 : 8, out var match) || + !MakeRuntimeBufferSource(match.TableHandle, pc, out var tableSourceIndex, out var tableSource)) + return false; + + var imageSource = new DescriptorSource + { + Dwords = [.. tableSource.Dwords, .. tableSource.Dwords], + IndirectImage = new IndirectImageSelector(0, tableSourceIndex, 0, 0, 0) + { + Dense = true, + TableOffset = match.TableOffset, + BufferTableStride = match.Stride, + PointerTargetOffset = match.TargetOffset, + }, + }; + + plan = new IndirectImagePlan + { + Handle = handle, + Source = InternSource(imageSource), + Key = match.KeyRead, + KeyIsAddressOffset = true, + HeapSource = tableSourceIndex, + SuppressMemoryReads = match.Reads.All(read => UsesOnly(read, [handle])), + Memory = match.MemoryIndices, + Reads = match.Reads, + }; + return true; + } + + // A sampler reached the same way. The host binds one sampler, so the table's records + // must all hold the same one; the materializer checks that on every snapshot. + private bool TryMakePointerTableSampler(ScalarValue? handle, uint pc, out uint sourceIndex) + { + sourceIndex = DescriptorConstants.NoIndex; + if (handle is not { Kind: ScalarValueKind.SamplerHandle } || !TryMatchPointerTable(handle, 4, out var match) || + !MakeRuntimeBufferSource(match.TableHandle, pc, out _, out var tableSource)) + return false; + + sourceIndex = InternSource(new DescriptorSource + { + Dwords = tableSource.Dwords, + PointerTable = new PointerTableSelector(match.TableOffset, match.Stride, match.TargetOffset), + }); + return true; + } + + // The record offset is key * stride, as a multiply or a shift by a constant. + private static bool TryGetRecordStride(ScalarValue offset, out uint stride) + { + stride = 0; + if (offset.Kind != ScalarValueKind.Operation || offset.Operands.Length != 2) + return false; + + if (offset.Operation == ScalarOperation.IMul32) + { + if (offset.Operands[1].IsConstant) + stride = offset.Operands[1].ConstantU32; + else if (offset.Operands[0].IsConstant) + stride = offset.Operands[0].ConstantU32; + } + else if (offset.Operation == ScalarOperation.ShiftLeft32 && offset.Operands[1].IsConstant && offset.Operands[1].ConstantU32 < 32) + { + stride = 1u << (int)offset.Operands[1].ConstantU32; + } + + return stride >= 8 * sizeof(uint) && stride % sizeof(uint) == 0; + } } diff --git a/src/SharpEmu.ShaderCompiler/Resources/ResourceTracker.cs b/src/SharpEmu.ShaderCompiler/Resources/ResourceTracker.cs index d96a20f..9105b6d 100644 --- a/src/SharpEmu.ShaderCompiler/Resources/ResourceTracker.cs +++ b/src/SharpEmu.ShaderCompiler/Resources/ResourceTracker.cs @@ -11,7 +11,6 @@ namespace SharpEmu.ShaderCompiler.Resources; public sealed partial class ResourceTracker { private const uint SamplerBorderClampMask = (1u << 2) | (1u << 5) | (1u << 8); - private const uint SamplerDword3ReservedMask = 0x3FFF_F000u; private readonly ShaderResourcePlan _plan; private readonly ScalarValueGraph _graph; @@ -276,8 +275,8 @@ public sealed partial class ResourceTracker var left = value.Operands[0]; var right = value.Operands[1]; - var leftReserved = (PossibleBits(left) & ~SamplerDword3ReservedMask) == 0; - var rightReserved = (PossibleBits(right) & ~SamplerDword3ReservedMask) == 0; + var leftReserved = (PossibleBits(left) & ~DescriptorConstants.SamplerDword3ReservedMask) == 0; + var rightReserved = (PossibleBits(right) & ~DescriptorConstants.SamplerDword3ReservedMask) == 0; if (leftReserved && rightReserved) { return _graph.Constant(0u); @@ -325,7 +324,9 @@ public sealed partial class ResourceTracker for (var candidate = 0; candidate < _sources.Count; candidate++) { var current = _sources[candidate]; - if (current.DwordCount != source.DwordCount || !Equals(current.IndirectImage, source.IndirectImage)) + if (current.DwordCount != source.DwordCount || !Equals(current.IndirectImage, source.IndirectImage) || + !Equals(current.PointerTable, source.PointerTable) || + current.ZeroExtentBufferSource != source.ZeroExtentBufferSource) { continue; } @@ -361,6 +362,7 @@ public sealed partial class ResourceTracker { var current = _bufferCandidateTables[existing]; if (!_graph.Equivalent(current.SrtHandle, table.SrtHandle) || + current.BaseOffset != table.BaseOffset || !_graph.Equivalent(current.OffsetExpression, table.OffsetExpression)) { continue; @@ -384,7 +386,8 @@ public sealed partial class ResourceTracker private bool IsHostBufferHandle(ScalarValue? handle) => handle is { Kind: ScalarValueKind.BufferHandle, Operands.Length: 4 } && - handle.Operands.All(dword => dword.Type == ScalarValueType.U32 && _plan.ValidateRuntimeValue(dword)); + handle.Operands.All(dword => dword.Type == ScalarValueType.U32 && _plan.ValidateRuntimeValue(dword) && + !DependsOnLoopCarriedRead(dword)); private bool IsDeviceLoadedBufferHandle(ScalarValue? handle) => handle is { Kind: ScalarValueKind.BufferHandle, Operands.Length: 4 } && @@ -419,6 +422,56 @@ public sealed partial class ResourceTracker return false; } + // A value that a loop recomputes from memory it read on the previous iteration (a + // pointer walk) differs per iteration even when its phis look structurally invariant. + private bool DependsOnLoopCarriedRead(ScalarValue value) + { + var phis = new HashSet(); + var pending = new Stack(); + var visited = new HashSet(); + pending.Push(value); + while (pending.TryPop(out var current)) + { + if (!visited.Add(current)) + continue; + if (current.Kind == ScalarValueKind.Phi) + phis.Add(current); + foreach (var operand in Dependencies(current)) + pending.Push(operand); + } + + foreach (var phi in phis) + { + // Look for the phi again below a memory read reached from its own operands. + var stack = new Stack<(ScalarValue Value, bool ThroughRead)>(); + var seen = new HashSet<(ScalarValue, bool)>(); + foreach (var operand in phi.Operands) + stack.Push((operand, false)); + while (stack.TryPop(out var entry)) + { + if (!seen.Add(entry)) + continue; + if (ReferenceEquals(entry.Value, phi) && entry.ThroughRead) + return true; + if (ReferenceEquals(entry.Value, phi)) + continue; + var throughRead = entry.ThroughRead || + entry.Value.Kind is ScalarValueKind.ScalarAddressWord or ScalarValueKind.ScalarBufferWord or + ScalarValueKind.ResourceTableWord; + foreach (var operand in Dependencies(entry.Value)) + stack.Push((operand, throughRead)); + } + } + + return false; + } + + // A resource-table word stands for the flattened read it names. + private IEnumerable Dependencies(ScalarValue value) => + value.Kind == ScalarValueKind.ResourceTableWord && value.Payload < (ulong)_plan.TableReads.Count + ? [_plan.TableReads[(int)value.Payload].Value] + : value.Operands; + private uint GetHandleSource( ScalarValue? handle, ScalarValueKind expected, @@ -427,7 +480,8 @@ public sealed partial class ResourceTracker bool sampler = false, bool sampleAdjust = false, string? memoryOpcode = null, - MemoryAccess memoryAccess = MemoryAccess.Read) + MemoryAccess memoryAccess = MemoryAccess.Read, + bool allowNullFallback = true) { if (handle is null || handle.Kind != expected) { @@ -448,6 +502,12 @@ public sealed partial class ResourceTracker var controlDependent = false; if (nonContiguousImage || !ValidateSource(source, out badDword, out controlDependent)) { + // Words read from an empty scalar buffer are zero whatever the offset. An access the + // runtime table can read keeps that path, which also covers a buffer that is not empty. + if (allowNullFallback && expected is ScalarValueKind.ImageHandle or ScalarValueKind.SamplerHandle && + TryMakeZeroExtentBufferSource(source, pc, out var emptyBufferSource)) + return emptyBufferSource; + // A bindless image/sampler descriptor whose dwords resolve through a // control-dependent phi (e.g. a hash-table/linear-probe material lookup, as seen // in Ghost of Yotei) has no single compile-time source: real support needs @@ -457,11 +517,12 @@ public sealed partial class ResourceTracker // mirroring KytyPS5's fallback for the same case (feat/shader-control-dependent- // descriptor). Buffer/sampler-adjacent handles or any other validation failure // still hard-fail, since those aren't safe to silently zero. - var dynamicImageFallback = expected is (ScalarValueKind.ImageHandle or ScalarValueKind.SamplerHandle) && + var dynamicImageFallback = allowNullFallback && expected is (ScalarValueKind.ImageHandle or ScalarValueKind.SamplerHandle) && (controlDependent || HasUndefinedOrigin(source.Dwords[badDword], "BufferLoadFormat") || (nonContiguousImage && source.Dwords.Any(dword => HasUndefinedOrigin(dword, "SAndB32")))); if (dynamicImageFallback) { + _info.NullDescriptorFallbacks.Add((pc, expected == ScalarValueKind.ImageHandle ? $"image {memoryOpcode ?? "access"}" : $"sampler {memoryOpcode ?? "access"}")); source = new DescriptorSource { Dwords = Enumerable.Repeat(_graph.Constant(0u), (int)source.DwordCount).ToArray(), @@ -480,6 +541,37 @@ public sealed partial class ResourceTracker return InternSource(source); } + private bool TryMakeZeroExtentBufferSource(DescriptorSource source, uint pc, out uint sourceIndex) + { + sourceIndex = 0; + ScalarValue? bufferHandle = null; + var hasBufferRead = false; + foreach (var operand in source.Dwords) + { + var word = _graph.ResolveInvariantPhi(operand); + if (word is null) return false; + if (word.IsConstant && word.ConstantU32 == 0) continue; + if (word.Kind != ScalarValueKind.ScalarBufferWord || word.Operands.Length != 2) + return false; + var current = word.Operands[0]; + if (bufferHandle is not null && !_graph.Equivalent(bufferHandle, current)) + return false; + bufferHandle = current; + hasBufferRead = true; + } + + if (!hasBufferRead || bufferHandle is null || + !MakeRuntimeBufferSource(bufferHandle, pc, out var bufferSource, out _)) + return false; + + sourceIndex = InternSource(new DescriptorSource + { + Dwords = Enumerable.Repeat(_graph.Constant(0u), (int)source.DwordCount).ToArray(), + ZeroExtentBufferSource = bufferSource, + }); + return true; + } + private string DescribeValueShape(ScalarValue value, int depth = 0) { if (depth >= 4) @@ -507,7 +599,7 @@ public sealed partial class ResourceTracker return $"{value.Kind}#{value.Id}"; } - // An image descriptor read from a scalar buffer must be one contiguous record. + // An image descriptor read from a scalar buffer must be made of contiguous four-dword runs. private bool IsContiguousScalarBufferRecord(DescriptorSource source) { if (!source.Dwords.Any(dword => dword.Kind == ScalarValueKind.ScalarBufferWord)) @@ -515,15 +607,17 @@ public sealed partial class ResourceTracker return true; } - var first = ScalarReadMemory(source.Dwords[0], out _); + // Each dword is evaluated on its own, so a record may be assembled from several + // four-dword loads of the same buffer (an 8-dword T# built by two s_buffer_load_x4). for (var dword = 0; dword < source.Dwords.Length; dword++) { var read = source.Dwords[dword]; + var first = ScalarReadMemory(source.Dwords[dword & ~3], out _); var memory = ScalarReadMemory(read, out _); if (first is null || memory is null || - memory.Offset != first.Offset + (uint)dword * sizeof(uint) || - !_graph.Equivalent(read.Operands[0], source.Dwords[0].Operands[0]) || - !_graph.Equivalent(read.Operands[1], source.Dwords[0].Operands[1])) + memory.Offset != first.Offset + (uint)(dword & 3) * sizeof(uint) || + !_graph.Equivalent(read.Operands[0], source.Dwords[dword & ~3].Operands[0]) || + !_graph.Equivalent(read.Operands[1], source.Dwords[dword & ~3].Operands[1])) { return false; } @@ -640,43 +734,27 @@ public sealed partial class ResourceTracker private static bool HasRuntimeReadKind(ScalarValue value, ScalarValueKind kind) => HasRuntimeReadKind(value, kind, []); - private static bool HasRuntimeReadKind(ScalarValue value, ScalarValueKind kind, HashSet visiting) + // Reachability: a node already explored cannot add a new answer, so the visited set is + // kept for the whole walk (removing it on return re-walks shared subgraphs exponentially). + private static bool HasRuntimeReadKind(ScalarValue value, ScalarValueKind kind, HashSet visited) { - if (!visiting.Add(value)) + if (!visited.Add(value)) { return false; } - try - { - return value.Kind == kind || value.Operands.Any(operand => HasRuntimeReadKind(operand, kind, visiting)); - } - finally - { - visiting.Remove(value); - } + return value.Kind == kind || value.Operands.Any(operand => HasRuntimeReadKind(operand, kind, visited)); } - private static bool HasRuntimeRead(ScalarValue value, HashSet visiting) + private static bool HasRuntimeRead(ScalarValue value, HashSet visited) { - if (!visiting.Add(value)) + if (!visited.Add(value)) { return false; } - try - { - if (value.Kind is ScalarValueKind.ScalarAddressWord or ScalarValueKind.ScalarBufferWord) - { - return true; - } - - return value.Operands.Any(operand => HasRuntimeRead(operand, visiting)); - } - finally - { - visiting.Remove(value); - } + return value.Kind is ScalarValueKind.ScalarAddressWord or ScalarValueKind.ScalarBufferWord || + value.Operands.Any(operand => HasRuntimeRead(operand, visited)); } // ---- dense tables ---- @@ -722,6 +800,8 @@ public sealed partial class ResourceTracker resource.Atomic |= atomic; resource.Formatted |= memory.Formatted; resource.Scalar |= memory.Kind == MemoryResourceKind.ScalarBuffer; + resource.DwordAddressed |= atomic || memory.Kind == MemoryResourceKind.ScalarBuffer || + (!memory.Formatted && !memory.Typed && memory.DataBits >= 32); } private uint AddImage(uint source, MemoryAccessInfo memory, uint pc) @@ -954,10 +1034,26 @@ public sealed partial class ResourceTracker return; } - // Scalar loads can address buffers that have no host descriptor binding, while - // vector buffer descriptors loaded from scalar-buffer data must stay device-side. + // Scalar loads can address buffers that have no host descriptor binding. A vector + // buffer descriptor loaded from scalar-buffer data is bound like any other when + // the draw can evaluate it (the V# is read from guest memory at dispatch); only one + // that stays unknown at dispatch reads through its registers on the device. if ((memory.Kind == MemoryResourceKind.ScalarBuffer && !IsHostBufferHandle(access.Handle)) || - (memory.Kind == MemoryResourceKind.Buffer && IsDeviceLoadedBufferHandle(access.Handle))) + (memory.Kind == MemoryResourceKind.Buffer && IsDeviceLoadedBufferHandle(access.Handle) && + !IsHostBufferHandle(access.Handle))) + { + memory.DeviceDescriptor = true; + _info.UsesDeviceAddresses = true; + return; + } + + // A V# chosen by control flow (a phi over several descriptors) has no single source + // to bind. Its SGPRs hold the selected descriptor when the access runs, so the + // access reads it there, as the hardware does, through the device-address table. + if (memory.Kind == MemoryResourceKind.Buffer && access.Handle is { Kind: ScalarValueKind.BufferHandle, Operands.Length: 4 } && + !memory.Opcode.StartsWith("TBuffer", StringComparison.Ordinal) && + ((!ValidateSource(MakeSource(access.Handle, 4, false, false, memory.Pc), out _, out var controlDependent) && controlDependent) || + access.Handle.Operands.Any(DependsOnLoopCarriedRead))) { memory.DeviceDescriptor = true; _info.UsesDeviceAddresses = true; @@ -1042,14 +1138,36 @@ public sealed partial class ResourceTracker } uint imageSource; - var indirect = _indirectImages.FirstOrDefault(plan => ReferenceEquals(plan.Handle, access.Handle)); - if (indirect is not null) + uint samplerSource = 0; + // The hardware reads both descriptors from memory when the access runs. An access the + // runtime descriptor table can serve does the same instead of reading the null descriptor. + var runtimeReadable = CanReadDescriptorsAtRuntime(memory, access); + try { - imageSource = indirect.Source; + var indirect = _indirectImages.FirstOrDefault(plan => ReferenceEquals(plan.Handle, access.Handle)); + imageSource = indirect is not null + ? indirect.Source + : GetHandleSource(access.Handle, ScalarValueKind.ImageHandle, 8, memory.Pc, allowNullFallback: !runtimeReadable); + if (memory.NeedsSampler) + { + if (access.SamplerHandle is null) + { + throw Failure(memory.Pc, "sampled image operation has no sampler handle"); + } + + var sampleAdjust = (memory.ImageSampleFlags & ImageSampleFlags.Adjust) != 0; + samplerSource = !sampleAdjust && TryMakePointerTableSampler(access.SamplerHandle, memory.Pc, out var pointerSampler) + ? pointerSampler + : GetHandleSource(access.SamplerHandle, ScalarValueKind.SamplerHandle, 4, memory.Pc, sampler: true, sampleAdjust, + allowNullFallback: !runtimeReadable); + } } - else + catch (ResourcePlanException) when (runtimeReadable) { - imageSource = GetHandleSource(access.Handle, ScalarValueKind.ImageHandle, 8, memory.Pc); + // No plan-time source: the shader reads both descriptors from its registers. + memory.RuntimeDescriptor = true; + _info.UsesRuntimeDescriptors = true; + return; } var image = AddImage(imageSource, memory, memory.Pc); @@ -1061,13 +1179,6 @@ public sealed partial class ResourceTracker uint sampler = 0; if (memory.NeedsSampler) { - if (access.SamplerHandle is null) - { - throw Failure(memory.Pc, "sampled image operation has no sampler handle"); - } - - var sampleAdjust = (memory.ImageSampleFlags & ImageSampleFlags.Adjust) != 0; - var samplerSource = GetHandleSource(access.SamplerHandle, ScalarValueKind.SamplerHandle, 4, memory.Pc, sampler: true, sampleAdjust); sampler = AddSampler(samplerSource, memory.Pc); if (sampler == DescriptorConstants.NoIndex) { @@ -1080,6 +1191,19 @@ public sealed partial class ResourceTracker AddMemoryPatch(index, image, sampler, memory.NeedsSampler, memory.Pc); } + // A sampled access or an image load that the host can create from descriptor words alone. + // An adjusted sample keys its sampler without the reserved bits its adjustment uses. Depth + // compares, cube views, storage writes and atomics keep their planned path. + private bool CanReadDescriptorsAtRuntime(MemoryAccessInfo memory, MemoryAccessBinding access) => + memory.ImageClass == ImageResourceClass.Sampled && + ((memory.NeedsSampler && access.SamplerHandle is not null && + (memory.ImageSampleFlags & ImageSampleFlags.Compare) == 0) || + (!memory.NeedsSampler && memory.Opcode is "ImageLoad" or "ImageLoadMip")) && + RuntimeDescriptorTable.SupportsRuntimeView(memory.ImageDimension) && + _graph.Program.Instructions.FirstOrDefault(instruction => instruction.Pc == memory.Pc)?.Control is Gen5ImageControl { Dimension: not CubeDimension }; + + private const uint CubeDimension = 3; + private void LinkImageAliases() { foreach (var buffer in _info.Buffers) @@ -1152,14 +1276,19 @@ public sealed partial class ResourceTracker for (var index = 0; index < _plan.Memory.Count; index++) { var memory = _plan.Memory[index]; - if (memory.Kind != MemoryResourceKind.Image || _plan.Accesses[index]?.Handle is not { } handle || + if (memory.Kind != MemoryResourceKind.Image || _plan.Accesses[index] is not { Handle: { } handle } access || _indirectImages.Any(plan => ReferenceEquals(plan.Handle, handle))) { continue; } + // A table in a buffer materializes every record it holds, and each draw binds them all: + // thousands for a material table. An access the runtime table can read binds only the + // records the shader actually reads. if (TryMakeIndirectImage(handle, memory.Pc, out var plan) || TryMakeDenseIndirectImage(handle, memory.Pc, out plan) || + (!CanReadDescriptorsAtRuntime(memory, access) && TryMakeBufferTableImage(handle, memory.Pc, memory.ImageR128, out plan)) || + TryMakePointerTableImage(handle, memory.Pc, memory.ImageR128, out plan) || TryMakeDirectImage(handle, out plan)) { _indirectImages.Add(plan); diff --git a/src/SharpEmu.ShaderCompiler/Resources/RuntimeDescriptorTable.cs b/src/SharpEmu.ShaderCompiler/Resources/RuntimeDescriptorTable.cs new file mode 100644 index 0000000..da90a5e --- /dev/null +++ b/src/SharpEmu.ShaderCompiler/Resources/RuntimeDescriptorTable.cs @@ -0,0 +1,163 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +namespace SharpEmu.ShaderCompiler.Resources; + +// The contract between a shader that reads its image and sampler descriptors at run time +// (MemoryAccessInfo.RuntimeDescriptor) and the host that resolves them. +// +// A waterfall loop or a per-pixel material pointer gives a descriptor no plan-time source, +// so the shader looks its words up in a global hash table the host fills: the image or +// sampler words map to a slot of the persistent bindless heap. A descriptor the table does +// not hold yet reads slot 0 (the null descriptor) and is appended to the miss buffer; the +// host creates it after the tick and inserts it, so a later draw finds it. +// +// Table buffer (dwords): +// [0] image capacity (entries, a power of two, 0 when empty) +// [1] sampler capacity (entries, a power of two, 0 when empty) +// [2] first dword of the image entries +// [3] first dword of the sampler entries +// image entry: slot + 1 (0 = empty), view class, image words 0..7 +// sampler entry: slot + 1 (0 = empty), sampler words 0..3 +// Miss buffer (dwords): +// [0] image misses written, [1] sampler misses written (atomic counters, may exceed the capacity) +// from ImageMissOffset: ImageMissCapacity records of (view class, image words 0..7) +// from SamplerMissOffset: SamplerMissCapacity records of sampler words 0..3 +public static class RuntimeDescriptorTable +{ + public const uint ImageWordCount = 8; + public const uint SamplerWordCount = 4; + + public const uint ImageCapacityDword = 0; + public const uint SamplerCapacityDword = 1; + public const uint ImageEntriesDword = 2; + public const uint SamplerEntriesDword = 3; + public const uint HeaderDwords = 4; + + public const uint ImageEntryDwords = 2 + ImageWordCount; + public const uint SamplerEntryDwords = 1 + SamplerWordCount; + + // Probes a lookup makes before it treats the descriptor as missing. The host never + // places an entry further than this from its home position. + public const uint ProbeCount = 8; + + public const uint ImageMissCountDword = 0; + public const uint SamplerMissCountDword = 1; + public const uint ImageMissCapacity = 64; + public const uint SamplerMissCapacity = 64; + public const uint ImageMissRecordDwords = 1 + ImageWordCount; + public const uint SamplerMissRecordDwords = SamplerWordCount; + public const uint ImageMissOffset = 4; + public const uint SamplerMissOffset = ImageMissOffset + ImageMissCapacity * ImageMissRecordDwords; + public const uint MissBufferDwords = SamplerMissOffset + SamplerMissCapacity * SamplerMissRecordDwords; + + // FNV-1a over the key dwords; the shader computes the same value. + public const uint HashSeed = 0x811C9DC5; + public const uint HashPrime = 0x01000193; + + public static uint Hash(ReadOnlySpan words) + { + var hash = HashSeed; + foreach (var word in words) + { + hash = unchecked((hash ^ word) * HashPrime); + } + + return hash; + } + + // The view a runtime-descriptor access declares. Float views deliberately retain the + // old dimension-only value so existing sampled-descriptor keys stay unchanged; integer + // views occupy the next two 8-bit ranges. + public const uint ViewClassNumericShift = 8; + private const uint ViewClassDimensionMask = (1u << (int)ViewClassNumericShift) - 1; + + public static uint ViewClass(ImageDimension dimension) => ViewClass(dimension, ImageNumericClass.Float); + + public static uint ViewClass(ImageDimension dimension, ImageNumericClass numericClass) => + (uint)dimension | (NumericClassCode(numericClass) << (int)ViewClassNumericShift); + + public static ImageDimension ViewDimension(uint viewClass) => + (ImageDimension)(viewClass & ViewClassDimensionMask); + + public static ImageNumericClass ViewNumericClass(uint viewClass) => + ((viewClass >> (int)ViewClassNumericShift) & 0xff) switch + { + 1 => ImageNumericClass.Uint, + 2 => ImageNumericClass.Sint, + _ => ImageNumericClass.Float, + }; + + public static uint NumericClassCode(ImageNumericClass numericClass) => numericClass switch + { + ImageNumericClass.Uint => 1, + ImageNumericClass.Sint => 2, + _ => 0, + }; + + public static bool SupportsRuntimeView(ImageDimension dimension) => + dimension is ImageDimension.Dim1D or ImageDimension.Dim1DArray or ImageDimension.Dim2D or + ImageDimension.Dim2DArray or ImageDimension.Dim3D; + + // The table words for the entries: every key sits within ProbeCount entries of its hash, + // and a section grows until all of its keys fit. + public static uint[] Build( + IReadOnlyList<(uint[] Key, uint Slot)> images, + IReadOnlyList<(uint[] Key, uint Slot)> samplers) + { + var imageSection = Place(images, ImageEntryDwords); + var samplerSection = Place(samplers, SamplerEntryDwords); + var imageCapacity = (uint)(imageSection.Length / ImageEntryDwords); + var samplerCapacity = (uint)(samplerSection.Length / SamplerEntryDwords); + var words = new uint[HeaderDwords + imageSection.Length + samplerSection.Length]; + words[ImageCapacityDword] = imageCapacity; + words[SamplerCapacityDword] = samplerCapacity; + words[ImageEntriesDword] = HeaderDwords; + words[SamplerEntriesDword] = HeaderDwords + (uint)imageSection.Length; + imageSection.CopyTo(words, (int)HeaderDwords); + samplerSection.CopyTo(words, (int)(HeaderDwords + imageSection.Length)); + return words; + } + + private static uint[] Place(IReadOnlyList<(uint[] Key, uint Slot)> entries, uint entryDwords) + { + if (entries.Count == 0) + { + return []; + } + + for (var capacity = (uint)Math.Max(16, (int)System.Numerics.BitOperations.RoundUpToPowerOf2((uint)entries.Count * 2)); ; capacity *= 2) + { + var section = new uint[capacity * entryDwords]; + var placed = true; + foreach (var (key, slot) in entries) + { + var hash = Hash(key); + var found = false; + for (uint probe = 0; probe < ProbeCount && !found; probe++) + { + var entry = ((hash + probe) & (capacity - 1)) * entryDwords; + if (section[entry] != 0) + { + continue; + } + + section[entry] = slot + 1; + key.CopyTo(section, (int)entry + 1); + found = true; + } + + if (!found) + { + placed = false; + break; + } + } + + if (placed) + { + return section; + } + } + } +} diff --git a/src/SharpEmu.ShaderCompiler/Resources/RuntimeValueEvaluator.cs b/src/SharpEmu.ShaderCompiler/Resources/RuntimeValueEvaluator.cs index b368b36..68e3808 100644 --- a/src/SharpEmu.ShaderCompiler/Resources/RuntimeValueEvaluator.cs +++ b/src/SharpEmu.ShaderCompiler/Resources/RuntimeValueEvaluator.cs @@ -148,8 +148,23 @@ public sealed class RuntimeValueEvaluator return true; case ScalarValueKind.Phi: { - var invariant = _plan.Graph.ResolveInvariantPhi(value); - return invariant is not null && EvaluateWide(invariant, out result); + // Every leaf of an invariant phi is the same value; one may still be mid- + // evaluation higher up (a loop-carried register), so any leaf that evaluates + // gives the result. + if (_plan.Graph.ResolveInvariantPhi(value) is null) + { + return false; + } + + foreach (var leaf in PhiLeaves(value)) + { + if (EvaluateWide(leaf, out result)) + { + return true; + } + } + + return false; } case ScalarValueKind.FirstLane: { @@ -195,6 +210,32 @@ public sealed class RuntimeValueEvaluator } } + private static List PhiLeaves(ScalarValue phi) + { + var leaves = new List(); + var pending = new Stack(); + var visited = new HashSet(); + pending.Push(phi); + while (pending.TryPop(out var current)) + { + if (current.Kind != ScalarValueKind.Phi) + { + leaves.Add(current); + continue; + } + + if (visited.Add(current)) + { + foreach (var operand in current.Operands) + { + pending.Push(operand); + } + } + } + + return leaves; + } + // A raw read adds the immediate and dynamic offsets to the 48-bit handle base, checks // a buffer read against its records, and reads one aligned dword. private bool EvaluateRawRead(ScalarValue value, out ulong result) @@ -282,8 +323,8 @@ public sealed class RuntimeValueEvaluator var size = stride == 0 ? (ulong)(uint)records : (ulong)stride * (uint)records; if (aligned > size || size - aligned < sizeof(uint)) { - // An unbound (empty) V# reads as zero; overrunning a bound buffer stays a failure. - return (uint)records == 0 ? RawAddress.Zero : RawAddress.Failed; + // Scalar loads outside the descriptor range return zero. + return RawAddress.Zero; } address = ((baseAddress & ~3ul) + byteOffset) & ~3ul; diff --git a/src/SharpEmu.ShaderCompiler/Resources/ScalarGraphDiskCache.cs b/src/SharpEmu.ShaderCompiler/Resources/ScalarGraphDiskCache.cs index 4f8e10f..826bf14 100644 --- a/src/SharpEmu.ShaderCompiler/Resources/ScalarGraphDiskCache.cs +++ b/src/SharpEmu.ShaderCompiler/Resources/ScalarGraphDiskCache.cs @@ -62,14 +62,7 @@ internal static class ScalarGraphDiskCache Directory.CreateDirectory(directory); // Keep a bounded cache across compiler versions too. No live // analysis objects or guest memory are retained by the cache. - var entries = new DirectoryInfo(directory).GetFiles("*.graph").OrderBy(f => f.LastWriteTimeUtc).ToArray(); - var total = entries.Sum(f => f.Length) + stream.Length + 32; - foreach (var entry in entries) - { - if (total <= MaxCacheBytes) break; - total -= entry.Length; - entry.Delete(); - } + EvictOldest(directory, stream.Length + 32); var temporary = path + "." + Guid.NewGuid().ToString("N") + ".tmp"; try { @@ -79,6 +72,8 @@ internal static class ScalarGraphDiskCache file.Write(bytes); } File.Move(temporary, path, true); + _entries!.Enqueue((path, stream.Length + 32)); + _totalBytes += stream.Length + 32; } finally { File.Delete(temporary); } } @@ -89,6 +84,30 @@ internal static class ScalarGraphDiskCache return graph; } + // The entries of the cache directory, oldest first, and their total size. The directory is + // listed once per process: a new build misses on every shader, and listing it per miss cost + // more than the analysis it saved. Caller holds WriteLock. + private static string? _entriesDirectory; + private static Queue<(string Path, long Bytes)>? _entries; + private static long _totalBytes; + + private static void EvictOldest(string directory, long incoming) + { + if (_entries is null || _entriesDirectory != directory) + { + var files = new DirectoryInfo(directory).GetFiles("*.graph").OrderBy(f => f.LastWriteTimeUtc).ToArray(); + _entries = new Queue<(string Path, long Bytes)>(files.Select(f => (f.FullName, f.Length))); + _totalBytes = files.Sum(f => f.Length); + _entriesDirectory = directory; + } + + while (_totalBytes + incoming > MaxCacheBytes && _entries.TryDequeue(out var entry)) + { + _totalBytes -= entry.Bytes; + File.Delete(entry.Path); + } + } + internal static string Key(Gen5ShaderProgram program, uint userDataBase, uint userDataCount, IReadOnlySet? fixedFunctionVertexLoads, uint waveSize) { diff --git a/src/SharpEmu.ShaderCompiler/Resources/ScalarValue.cs b/src/SharpEmu.ShaderCompiler/Resources/ScalarValue.cs index 72211ce..750dfe3 100644 --- a/src/SharpEmu.ShaderCompiler/Resources/ScalarValue.cs +++ b/src/SharpEmu.ShaderCompiler/Resources/ScalarValue.cs @@ -365,6 +365,7 @@ public static class ScalarValueEquivalence } ScalarValue? invariant = null; + var leaves = new List(); var pending = new Stack(); var visited = new HashSet(); pending.Push(value); @@ -386,6 +387,7 @@ public static class ScalarValueEquivalence continue; } + leaves.Add(current); if (invariant is null) { invariant = current; @@ -396,6 +398,42 @@ public static class ScalarValueEquivalence } } + // A leaf computed from the phis it is merged into (a load whose address a loop + // advances) takes a new value on every iteration, however alike its nodes look. + foreach (var leaf in leaves) + { + if (DependsOnAny(leaf, visited)) + { + return null; + } + } + return invariant; } + + private static bool DependsOnAny(ScalarValue value, HashSet phis) + { + var pending = new Stack(); + var seen = new HashSet(); + pending.Push(value); + while (pending.TryPop(out var current)) + { + if (!seen.Add(current)) + { + continue; + } + + if (current.Kind == ScalarValueKind.Phi && phis.Contains(current)) + { + return true; + } + + foreach (var operand in current.Operands) + { + pending.Push(operand); + } + } + + return false; + } } diff --git a/src/SharpEmu.ShaderCompiler/Resources/ScalarValueGraph.Builder.cs b/src/SharpEmu.ShaderCompiler/Resources/ScalarValueGraph.Builder.cs index ff75f5f..8d5b233 100644 --- a/src/SharpEmu.ShaderCompiler/Resources/ScalarValueGraph.Builder.cs +++ b/src/SharpEmu.ShaderCompiler/Resources/ScalarValueGraph.Builder.cs @@ -426,7 +426,14 @@ public sealed partial class ScalarValueGraph { case Gen5ShaderEncoding.Sop1: case Gen5ShaderEncoding.Sop2: + ApplyScalarAlu(instruction, state); + return; case Gen5ShaderEncoding.Sopk: + if (instruction.Opcode.StartsWith("SCmpk", StringComparison.Ordinal)) + { + ApplyScalarCompareK(instruction, state); + return; + } ApplyScalarAlu(instruction, state); return; case Gen5ShaderEncoding.Sopc: @@ -484,10 +491,31 @@ public sealed partial class ScalarValueGraph return; case "SBitreplicateB64B32": { - var replicated = Read(instruction.Sources[0], state); - state.WriteScalar(destinationRegister, replicated); - state.WriteScalar(destinationRegister + 1, replicated); + // Each source bit i fills destination bits 2i and 2i+1. + var source = Read(instruction.Sources[0], state); + if (source.IsConstant) + { + state.WriteScalar(destinationRegister, _graph.Constant(ReplicateBits(source.ConstantU32 & 0xFFFF))); + state.WriteScalar(destinationRegister + 1, _graph.Constant(ReplicateBits(source.ConstantU32 >> 16))); + } + else + { + state.WriteScalar(destinationRegister, _graph.Undefined(ScalarValueType.U32)); + state.WriteScalar(destinationRegister + 1, _graph.Undefined(ScalarValueType.U32)); + } + return; + + static uint ReplicateBits(uint value) + { + var result = 0u; + for (var bit = 0; bit < 16; bit++) + { + result |= ((value >> bit) & 1u) * (3u << (2 * bit)); + } + + return result; + } } case "SMovkI32": state.WriteScalar(destinationRegister, _graph.Constant(unchecked((uint)(short)instruction.Sources[0].Value))); @@ -664,6 +692,17 @@ public sealed partial class ScalarValueGraph case "SFF1I32B32": state.WriteScalar(destinationRegister, _graph.FindLowestSetBit(left, instruction.Pc)); return; + case "SFlbitI32B32": + { + // RDNA returns -1 for zero; otherwise this is the number of + // zeros before the most significant set bit. + var high = Unary(ScalarOperation.FindHighestBit32, left); + var leading = Binary(ScalarOperation.ISub32, _graph.Constant(31u), high); + state.WriteScalar(destinationRegister, + _graph.Select(Bool(ScalarOperation.IEqual32, left, _graph.Constant(0u)), + _graph.Constant(uint.MaxValue), leading)); + return; + } case "SBitset0B32": { var bit = Binary(ScalarOperation.ShiftLeft32, _graph.Constant(1u), Binary(ScalarOperation.And32, left, _graph.Constant(31u))); @@ -1073,6 +1112,28 @@ public sealed partial class ScalarValueGraph state.Scc = Compare(opcode["SCmp".Length..], left, right); } + private void ApplyScalarCompareK(Gen5ShaderInstruction instruction, RegisterState state) + { + if (instruction.Destinations.Count != 1 || + instruction.Destinations[0] is not { Kind: Gen5OperandKind.ScalarRegister, Value: < ScalarRegisterCount } source || + instruction.Words.Count == 0) + { + state.Scc = _graph.Undefined(ScalarValueType.Bool); + return; + } + + var simm16 = instruction.Words[0] & 0xFFFF; + var immediate = instruction.Opcode.EndsWith("I32", StringComparison.Ordinal) && + !instruction.Opcode.EndsWith("U32", StringComparison.Ordinal) + ? unchecked((uint)(int)(short)simm16) + : simm16; + + // SOPK compare instructions encode the source SGPR in the SDST field. + // They update SCC and leave that SGPR unchanged. + state.Scc = Compare(instruction.Opcode["SCmp".Length..], + state.Scalars[source.Value], _graph.Constant(immediate)); + } + private ScalarValue Compare(string suffix, ScalarValue left, ScalarValue right) => suffix switch { "EqI32" or "EqU32" or "kEqI32" or "kEqU32" => Bool(ScalarOperation.IEqual32, left, right), @@ -1587,7 +1648,7 @@ public sealed partial class ScalarValueGraph state.WriteVector(destination.Value, _graph.Undefined(ScalarValueType.U32)); if (!lane.IsConstant) { - state.ClearLanes(destination.Value); + state.ClearLanes(destination.Value, masked: false); return; } @@ -1909,10 +1970,12 @@ public sealed partial class ScalarValueGraph return; } - Vectors[register] = value.IsUndefined ? value : _graph.Select(Exec, value, Vectors[register]); + Vectors[register] = Exec.IsConstant && !Exec.ConstantBool + ? Vectors[register] + : value.IsUndefined ? value : _graph.Select(Exec, value, Vectors[register]); } - public void ClearLanes(uint register) + public void ClearLanes(uint register, bool masked = true) { if (Lanes.Count == 0) { @@ -1921,6 +1984,22 @@ public sealed partial class ScalarValueGraph foreach (var key in Lanes.Keys.Where(key => key.Register == register).ToArray()) { + // Ordinary VGPR writes affect only lanes enabled by EXEC. Writelane + // with a dynamic index is unmasked and must invalidate every lane. + if (masked) + { + // Phi operands still change while the builder converges. Do not + // use the graph's post-build invariant cache here. + var maskValue = Scalars[key.Lane < 32 ? ExecLow : ExecHigh]; + var mask = maskValue.Kind == ScalarValueKind.Phi + ? ScalarValueEquivalence.ResolveInvariantPhi(_graph.Memory, maskValue) + : maskValue; + if (mask is { IsConstant: true } && + (mask.ConstantU32 & (1u << (int)(key.Lane & 31))) == 0) + { + continue; + } + } Lanes.Remove(key); } } diff --git a/src/SharpEmu.ShaderCompiler/Resources/ShaderResourceInfo.cs b/src/SharpEmu.ShaderCompiler/Resources/ShaderResourceInfo.cs index 9154e70..002a478 100644 --- a/src/SharpEmu.ShaderCompiler/Resources/ShaderResourceInfo.cs +++ b/src/SharpEmu.ShaderCompiler/Resources/ShaderResourceInfo.cs @@ -12,6 +12,7 @@ public enum ShaderStage : byte Vertex, Pixel, Compute, + TessellationEvaluation, } public enum ImageNumericClass : byte @@ -35,6 +36,9 @@ public static class DescriptorConstants public const uint IdentityImageSwizzle = 0xFAC; public const uint InvalidFormat = 0; public const uint NoIndex = uint.MaxValue; + // GFX10 S# dword 3 bits 12-29 are reserved. The adjusted sample forms carry their + // adjustment there; the sampler itself ignores them. + public const uint SamplerDword3ReservedMask = 0x3FFF_F000u; } // One buffer resource of a program and how the program uses it. @@ -53,6 +57,11 @@ public sealed class BufferResource public bool Formatted { get; set; } public bool Scalar { get; set; } + // Set when some access addresses whole dwords (raw dword, atomic or scalar), which the + // host resolves with the byte address divided by four. Formatted and sub-dword accesses + // assemble their bytes, so a buffer used only that way may start at any byte. + public bool DwordAddressed { get; set; } + public BufferResource Clone() => (BufferResource)MemberwiseClone(); } @@ -156,8 +165,14 @@ public sealed class BufferCandidateTableInfo // The dense resource tables of a program plus the facts the pipeline layout needs. public sealed class ShaderResourceInfo { - public const int MaxBuffers = 32; - public const int MaxImages = 64; + // Buffers past this read their V# from registers through the device-address table, + // which costs far more shader code than a binding. Stages past 32 descriptors bind + // from a descriptor-heap set instead of push descriptors; the device's per-stage limit + // is checked when the layout is made. + public const int MaxBuffers = 256; + // Bindless material tables select among many descriptors; the host binds each + // candidate, and the device's per-stage limits are checked when the layout is made. + public const int MaxImages = 65536; public const int MaxSamplers = 32; public const int MaxSampledPairs = 64; public const int NoScalarRegister = -1; @@ -175,6 +190,13 @@ public sealed class ShaderResourceInfo public bool HasBitwiseExclusiveOr { get; set; } public bool UsesDeviceAddresses { get; set; } + // Some sampled access reads its descriptors through the runtime descriptor table. + public bool UsesRuntimeDescriptors { get; set; } + + // Descriptor accesses with no plan-time source that read the null descriptor instead + // (their texels read as zero); the host reports them so the loss is visible. + public List<(uint Pc, string Kind)> NullDescriptorFallbacks { get; set; } = []; + public ShaderResourceInfo Clone() => new() { Buffers = Buffers.Select(buffer => buffer.Clone()).ToList(), @@ -189,6 +211,8 @@ public sealed class ShaderResourceInfo InstanceOffsetScalarRegister = InstanceOffsetScalarRegister, HasBitwiseExclusiveOr = HasBitwiseExclusiveOr, UsesDeviceAddresses = UsesDeviceAddresses, + UsesRuntimeDescriptors = UsesRuntimeDescriptors, + NullDescriptorFallbacks = [.. NullDescriptorFallbacks], }; } @@ -208,10 +232,22 @@ public sealed record IndirectImageSelector( public uint KeyBound { get; init; } public WaveIndexedImageSelector? WaveIndexed { get; init; } + // Non-zero when the descriptors live in a buffer table of records this many bytes apart: + // the shader's scalar buffer offset selects the record, and the table's V# bounds it. + public uint BufferTableStride { get; init; } + // The key read's immediate offset. The hardware adds it after the 32-bit selector offset, without wrapping. public uint MaterialImmediate { get; init; } + + // Set when each buffer-table record holds a 64-bit pointer (at TableOffset) and the + // descriptor lives at this byte offset from it, rather than inside the record. + public uint? PointerTargetOffset { get; init; } } +// A descriptor reached through a V# table of records that each hold a 64-bit pointer: the +// record's pointer sits at TableOffset, the descriptor at TargetOffset from that pointer. +public sealed record PointerTableSelector(uint TableOffset, uint Stride, uint TargetOffset); + public sealed record DirectImageCandidate(uint Offset, uint Source); // A wave-uniform descriptor selector. The guest derives each descriptor key from @@ -226,6 +262,12 @@ public sealed class DescriptorSource public ScalarValue[] Dwords { get; init; } = []; public uint DwordCount => (uint)Dwords.Length; public IndirectImageSelector? IndirectImage { get; init; } + + // For a sampler: Dwords are the table V#, and every record's sampler must be the same. + public PointerTableSelector? PointerTable { get; init; } + // Scalar loads from an empty buffer return zero regardless of the offset. + // Materialization must recheck the source extent before using these words. + public uint? ZeroExtentBufferSource { get; init; } } // One immediate-offset scalar read the host evaluates into the flattened table. @@ -240,3 +282,19 @@ public readonly record struct DescriptorWords(uint[] Dwords) public bool SameAs(DescriptorWords other) => Dwords.AsSpan().SequenceEqual(other.Dwords); } + +// Equality of descriptor dwords by content. +public sealed class DescriptorContentComparer : IEqualityComparer +{ + public static readonly DescriptorContentComparer Instance = new(); + + public bool Equals(uint[]? left, uint[]? right) => + ReferenceEquals(left, right) || (left is not null && right is not null && left.AsSpan().SequenceEqual(right)); + + public int GetHashCode(uint[] dwords) + { + var hash = new HashCode(); + hash.AddBytes(System.Runtime.InteropServices.MemoryMarshal.AsBytes(dwords.AsSpan())); + return hash.ToHashCode(); + } +} diff --git a/src/SharpEmu.ShaderCompiler/Resources/ShaderResourcePlan.cs b/src/SharpEmu.ShaderCompiler/Resources/ShaderResourcePlan.cs index 985b04f..63e700a 100644 --- a/src/SharpEmu.ShaderCompiler/Resources/ShaderResourcePlan.cs +++ b/src/SharpEmu.ShaderCompiler/Resources/ShaderResourcePlan.cs @@ -27,6 +27,10 @@ public sealed class ShaderResourcePlan public ScalarValueGraph Graph { get; } public ShaderStage Stage { get; } public ulong Hash { get; } + + // How often ResourceMaterializationCache looked this plan up and found a reusable entry. + internal int CacheLookups; + internal int CacheHits; public uint UserDataBase => Graph.UserDataBase; public uint UserDataCount => Graph.UserDataCount; public MemoryAccessTable Memory => Graph.Memory; @@ -148,6 +152,10 @@ public sealed class ShaderResourcePlan foreach (var sampler in plan.Info.Samplers) { materialization.Add(sampler.Source); + if (plan.DescriptorSources[(int)sampler.Source].PointerTable is not null) + { + plan.RequiresSpecializationMemory = true; + } } plan.MaterializationSources = materialization; diff --git a/src/SharpEmu.ShaderCompiler/ShaderCompileRequest.cs b/src/SharpEmu.ShaderCompiler/ShaderCompileRequest.cs index 2e64c02..cc6f5b1 100644 --- a/src/SharpEmu.ShaderCompiler/ShaderCompileRequest.cs +++ b/src/SharpEmu.ShaderCompiler/ShaderCompileRequest.cs @@ -130,6 +130,11 @@ public sealed class ShaderCompileRequest public Gen5ShaderProgram Program { get; } public ShaderStage Stage { get; } + public Gen5TessellationInfo? Tessellation { get; init; } + // Merged LS/HS workgroups can contain several independently branching waves. + // The Vulkan lowering rendezvous all waves before an LDS phase transition. + public Gen5TessellationHullInfo? TessellationHull { get; init; } + public bool CooperativeWave64Workgroup { get; init; } public ulong Hash { get; } public MemoryAccessTable Memory { get; } public SpecializedResourceInfo Resources { get; } @@ -171,11 +176,57 @@ public sealed class ShaderCompileRequest // 64-bit atomics are emitted as real 64-bit atomics instead of a pair of // 32-bit ones, which is not atomic as a pair. public bool SupportsSharedInt64Atomics { get; init; } + + // shaderFloat16 with round-to-nearest-even, denormal and signed zero/Inf/NaN + // preservation for f16 (float controls). Conversions and packed fused arithmetic + // may use native instructions; devices missing any guarantee use the software path. + public bool SupportsExactFloat16Conversions { get; init; } + + // Descriptor indexing with a per-invocation index (shader*ArrayNonUniformIndexing): an + // indirect image table is then read by index instead of one case per candidate. + public bool SupportsNonUniformImageIndexing { get; init; } + + // Emits the program-counter dispatcher even when the block graph can be structured, to + // compare the two control-flow forms. + public bool ForceDispatcher { get; init; } + + // Formatted buffer accesses whose format is known when the shader is translated still + // go through the run-time format decoder. For tests that compare the two. + public bool ForceGenericBufferFormats { get; init; } + + // True only when the host measured GLSL UnpackHalf2x16 / PackHalf2x16 to be bit-exact + // against the emulator's own f16 conversion on this device, for every half bit pattern + // and every rounding boundary. The packed-f16 paths then convert with those two ext + // instructions instead of the branchless integer sequences, which is what a driver + // lowers to one hardware convert. Default false: the spec leaves f16 rounding and + // subnormal behaviour to the implementation, so the exact emulation is the only safe + // default and native is opted into by a device probe, never by a capability bit. + public bool NativeHalfConversionExact { get; init; } + + // True only when the host measured that a storage-buffer read past the end of its descriptor + // range reads zero on this device (robustBufferAccess2 that the device actually honours). + // Every guest buffer word is then loaded directly: the range test, the address clamp and the + // zero select around it exist only to produce the zero the device already returns. Default + // false, because an out-of-range read is undefined without that promise. + public bool ZeroOutOfBoundsBufferReads { get; init; } public bool ShaderSignedZeroInfNanPreserveFloat32Supported { get; init; } public Gen5ComputeSystemRegisters? ComputeSystemRegisters { get; init; } public IReadOnlyList PixelOutputs { get; init; } = []; public uint PixelInputEnable { get; init; } + // Explicit interpolation sample without requiring a fragment invocation per sample. + // The caller must select a sample valid for the pipeline's rasterization count. + public uint? PixelInterpolationSample { get; init; } + // Offsets relative to pixel center over a row-major 2x2 grid. A fixed sample uses + // four offsets; per-sample interpolation uses rasterizationSamples offsets per pixel. + public IReadOnlyList<(float X, float Y)> PixelCustomSampleOffsets { get; init; } = []; + public bool EarlyFragmentTests { get; init; } + // Samples excluded from triggering guest fragment execution after early depth/stencil tests. + public uint PixelShaderSampleExclusionMask { get; init; } + public bool PixelDepthExportEnable { get; init; } + public bool PixelSampleMaskExportEnable { get; init; } + public uint PixelMaskExportSamples { get; init; } = 1; + public uint PixelRasterizationSamples { get; init; } = 1; public uint PixelCustomInterpolationMask { get; init; } // False when the device cannot read one vertex's value of a pixel input (PerVertexKHR). diff --git a/tests/SharpEmu.Libs.Tests/Agc/AgcCommandBufferChainTests.cs b/tests/SharpEmu.Libs.Tests/Agc/AgcCommandBufferChainTests.cs index c29336d..8b4a1f3 100644 --- a/tests/SharpEmu.Libs.Tests/Agc/AgcCommandBufferChainTests.cs +++ b/tests/SharpEmu.Libs.Tests/Agc/AgcCommandBufferChainTests.cs @@ -105,9 +105,11 @@ public sealed class AgcCommandBufferChainTests var ctx = new CpuContext(memory, Generation.Gen5); PointCommandBufferAt(memory, FirstLinkAddress); + // (acb, cachePolicy, target, sizeDwords), as Ghost of Yotei links its ACB blocks. ctx[CpuRegister.Rdi] = CommandBufferAddress; - ctx[CpuRegister.Rsi] = SecondLinkAddress; - ctx[CpuRegister.Rdx] = 0x123; + ctx[CpuRegister.Rsi] = 0; + ctx[CpuRegister.Rdx] = SecondLinkAddress; + ctx[CpuRegister.Rcx] = 0x123; Assert.Equal((int)OrbisGen2Result.ORBIS_GEN2_OK, AgcExports.AcbJump(ctx)); Assert.Equal(FirstLinkAddress, ctx[CpuRegister.Rax]); @@ -115,6 +117,22 @@ public sealed class AgcCommandBufferChainTests Assert.Equal(0x0F30_0123u, ReadUInt32(memory, FirstLinkAddress + 12)); } + [Fact] + public void AcbJump_EncodesCachePolicy() + { + var memory = new FakeCpuMemory(BaseAddress, MemorySize); + var ctx = new CpuContext(memory, Generation.Gen5); + + PointCommandBufferAt(memory, FirstLinkAddress); + ctx[CpuRegister.Rdi] = CommandBufferAddress; + ctx[CpuRegister.Rsi] = 2; + ctx[CpuRegister.Rdx] = SecondLinkAddress; + ctx[CpuRegister.Rcx] = 0x40; + + Assert.Equal((int)OrbisGen2Result.ORBIS_GEN2_OK, AgcExports.AcbJump(ctx)); + Assert.Equal(0x2F30_0040u, ReadUInt32(memory, FirstLinkAddress + 12)); + } + [Fact] public void SubmittedDcb_ReturnsToParentAfterIndirectCall() { diff --git a/tests/SharpEmu.Libs.Tests/Agc/AgcDirectDrawModifierTests.cs b/tests/SharpEmu.Libs.Tests/Agc/AgcDirectDrawModifierTests.cs new file mode 100644 index 0000000..6e015b7 --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Agc/AgcDirectDrawModifierTests.cs @@ -0,0 +1,90 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Buffers.Binary; +using SharpEmu.HLE; +using SharpEmu.Libs.Agc; +using SharpEmu.Libs.Gpu.GpuCommands; +using SharpEmu.Libs.Tests.Gpu.GpuCommands; +using Xunit; + +namespace SharpEmu.Libs.Tests.Agc; + +// Direct draws carry a modifier naming the shader stage and the user registers of the pipeline. +// Every modifier must still produce the draw: Unreal's volume passes use a geometry-stage modifier. +public sealed class AgcDirectDrawModifierTests +{ + private const ulong BaseAddress = 0x2_1000_0000; + private const ulong CommandBufferAddress = BaseAddress + 0x80; + private const ulong CommandAddress = BaseAddress + 0x200; + + public static TheoryData Modifiers => new() { 0x4000_0000UL, 0x6000_0000UL, 0xA000_0000UL, 0x4008_0205UL, 0x1_6000_0000UL }; + + [Theory] + [MemberData(nameof(Modifiers))] + public void DrawIndexAutoEmitsTheDrawForEveryModifier(ulong modifier) + { + var memory = CreateCommandBuffer(); + var context = new CpuContext(memory, Generation.Gen5); + context[CpuRegister.Rdi] = CommandBufferAddress; + context[CpuRegister.Rsi] = 6; + context[CpuRegister.Rdx] = modifier; + + AgcExports.DcbDrawIndexAuto(context); + + Assert.Equal(CommandAddress, context[CpuRegister.Rax]); + var runner = new StreamRunner(); + Assert.Equal(SubmissionProgress.Complete, runner.Run(ReadDwords(memory, CommandAddress, 7))); + var draw = Assert.Single(runner.Host.AutoDraws); + Assert.Equal(6u, draw.VertexCount); + } + + [Theory] + [MemberData(nameof(Modifiers))] + public void DrawIndexEmitsTheDrawForEveryModifier(ulong modifier) + { + var memory = CreateCommandBuffer(); + var context = new CpuContext(memory, Generation.Gen5); + context[CpuRegister.Rdi] = CommandBufferAddress; + context[CpuRegister.Rsi] = 36; + context[CpuRegister.Rdx] = StreamRunner.DataAddress; + context[CpuRegister.Rcx] = modifier; + + AgcExports.DcbDrawIndex(context); + + Assert.NotEqual(0UL, context[CpuRegister.Rax]); + var runner = new StreamRunner(); + Assert.Equal(SubmissionProgress.Complete, runner.Run(ReadDwords(memory, CommandAddress, 11))); + var draw = Assert.Single(runner.Host.IndexedDraws); + Assert.Equal(36u, draw.IndexCount); + Assert.Equal(StreamRunner.DataAddress, draw.IndexAddress); + } + + private static FakeCpuMemory CreateCommandBuffer() + { + var memory = new FakeCpuMemory(BaseAddress, 0x1000); + WriteUInt64(memory, CommandBufferAddress + 0x10, CommandAddress); + WriteUInt64(memory, CommandBufferAddress + 0x18, CommandAddress + 0x100); + return memory; + } + + private static uint[] ReadDwords(FakeCpuMemory memory, ulong address, int count) + { + var bytes = new byte[count * sizeof(uint)]; + Assert.True(memory.TryRead(address, bytes)); + var dwords = new uint[count]; + for (var index = 0; index < count; index++) + { + dwords[index] = BinaryPrimitives.ReadUInt32LittleEndian(bytes.AsSpan(index * sizeof(uint))); + } + + return dwords; + } + + private static void WriteUInt64(FakeCpuMemory memory, ulong address, ulong value) + { + Span buffer = stackalloc byte[sizeof(ulong)]; + BinaryPrimitives.WriteUInt64LittleEndian(buffer, value); + Assert.True(memory.TryWrite(address, buffer)); + } +} diff --git a/tests/SharpEmu.Libs.Tests/Agc/AgcEventWriteTests.cs b/tests/SharpEmu.Libs.Tests/Agc/AgcEventWriteTests.cs new file mode 100644 index 0000000..5e7a3cb --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Agc/AgcEventWriteTests.cs @@ -0,0 +1,77 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Buffers.Binary; +using SharpEmu.HLE; +using SharpEmu.Libs.Agc; +using SharpEmu.Libs.Gpu.GpuCommands; +using SharpEmu.Libs.Tests.Gpu.GpuCommands; +using Xunit; + +namespace SharpEmu.Libs.Tests.Agc; + +public sealed class AgcEventWriteTests +{ + [Fact] + public void DrawCounterDumpsPublishBothHalvesOfQueryResults() + { + var runner = new StreamRunner(); + var begin = Build(0x39, StreamRunner.LabelAddress); + var end = Build(0x39, StreamRunner.LabelAddress + 8); + + Assert.Equal(0x139u, begin[1]); + Assert.Equal(SubmissionProgress.Complete, runner.Run(begin, end)); + for (var block = 0u; block < 16; block++) + { + var address = StreamRunner.LabelAddress + block * 16; + var first = runner.Host.ReadQword(address); + var last = runner.Host.ReadQword(address + 8); + Assert.NotEqual(0UL, first & last & (1UL << 63)); + Assert.True(last > first); + } + } + + [Theory] + [InlineData(0x38u, 0x1234567800UL, 4)] + [InlineData(0x39u, 0x1234567800UL, 4)] + [InlineData(0x3Au, 0UL, 2)] + [InlineData(0x10u, 0UL, 2)] + public void DrawEventsPreservePayloadAndAdvanceCursor(uint eventType, ulong payload, int words) + { + var packet = Build(eventType, payload); + Assert.Equal(words, packet.Length); + Assert.Equal(eventType | (words == 4 ? 0x100u : 0), packet[1]); + if (words == 4) + { + Assert.Equal((uint)payload, packet[2]); + Assert.Equal((uint)(payload >> 32), packet[3]); + } + } + + private static uint[] Build(uint eventType, ulong payload) + { + const ulong memoryAddress = 0x2_1000_0000; + const ulong commandBuffer = memoryAddress + 0x80; + const ulong packetAddress = memoryAddress + 0x200; + var memory = new FakeCpuMemory(memoryAddress, 0x1000); + var context = new CpuContext(memory, Generation.Gen5); + Span bytes = stackalloc byte[16]; + BinaryPrimitives.WriteUInt64LittleEndian(bytes, packetAddress); + BinaryPrimitives.WriteUInt64LittleEndian(bytes[8..], packetAddress + 0x100); + Assert.True(memory.TryWrite(commandBuffer + 0x10, bytes)); + context[CpuRegister.Rdi] = commandBuffer; + context[CpuRegister.Rsi] = eventType; + context[CpuRegister.Rdx] = payload; + + AgcExports.DcbEventWrite(context); + Assert.Equal(packetAddress, context[CpuRegister.Rax]); + Assert.True(memory.TryRead(commandBuffer + 0x10, bytes[..8])); + var length = checked((int)(BinaryPrimitives.ReadUInt64LittleEndian(bytes) - packetAddress)); + Assert.InRange(length, 8, 16); + Assert.True(memory.TryRead(packetAddress, bytes[..length])); + var packet = new uint[length / 4]; + for (var i = 0; i < packet.Length; i++) + packet[i] = BinaryPrimitives.ReadUInt32LittleEndian(bytes[(i * 4)..]); + return packet; + } +} diff --git a/tests/SharpEmu.Libs.Tests/Agc/AgcFusedShaderTests.cs b/tests/SharpEmu.Libs.Tests/Agc/AgcFusedShaderTests.cs index 870744a..8e4589f 100644 --- a/tests/SharpEmu.Libs.Tests/Agc/AgcFusedShaderTests.cs +++ b/tests/SharpEmu.Libs.Tests/Agc/AgcFusedShaderTests.cs @@ -329,15 +329,17 @@ public sealed class AgcFusedShaderTests program.Instructions.Select(static instruction => instruction.Opcode)); } - [Fact] - public void EmbeddedFusedProgram_RegistersValidatedContinuationDescriptor() + [Theory] + [InlineData(false)] + [InlineData(true)] + public void EmbeddedFusedProgram_RegistersValidatedContinuationDescriptor(bool continuationBeforeEntry) { var memory = new FakeCpuMemory(BaseAddress, 0x1_0000); var ctx = new CpuContext(memory, Generation.Gen5); var entryHeader = BaseAddress + 0x100; var entryCode = BaseAddress + 0x1000; var continuationHeader = entryCode + 0x80; - var continuationCode = entryCode + 0x200; + var continuationCode = continuationBeforeEntry ? entryCode - 0x200 : entryCode + 0x200; WriteUInt32(memory, entryHeader, 0x34333231u); WriteUInt32(memory, entryHeader + sizeof(uint), 0x18u); diff --git a/tests/SharpEmu.Libs.Tests/Agc/AgcMarkerTests.cs b/tests/SharpEmu.Libs.Tests/Agc/AgcMarkerTests.cs new file mode 100644 index 0000000..daac32f --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Agc/AgcMarkerTests.cs @@ -0,0 +1,302 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Buffers.Binary; +using System.Reflection; +using SharpEmu.Core.Cpu; +using SharpEmu.HLE; +using SharpEmu.Libs.Agc; +using Xunit; + +namespace SharpEmu.Libs.Tests.Agc; + +// Restore marker packet reservations without allowing foreign markers to overwrite inline packets. +[Collection(AgcCommandBufferChainCollection.Name)] +public sealed class AgcMarkerTests +{ + private const ulong BaseAddress = 0x1_0000_0000; + private const int MemorySize = 0x4000; + private const ulong CommandBufferAddress = BaseAddress + 0x100; + private const ulong CursorAddress = BaseAddress + 0x1000; + private const ulong MarkerStringAddress = BaseAddress + 0x200; + private const uint PoisonDword = 0xDEAD_BEEFu; + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void Marker_ReservesReturnedStorageBeforeTheNextPacket(bool push) + { + var (memory, ctx) = Setup(); + ctx[CpuRegister.Rdi] = CommandBufferAddress; + ctx[CpuRegister.Rsi] = MarkerStringAddress; + if (push) AgcExports.DcbPushMarker(ctx); + else AgcExports.DcbPopMarker(ctx); + + var markerPacket = ctx[CpuRegister.Rax]; + WriteShaderRegister(ctx); + var registerPacket = ctx[CpuRegister.Rax]; + + // Patching a returned marker payload must not alter the following SET_SH_REG. + WriteUInt32(memory, markerPacket + 4, 0x74736574); + Assert.Equal(0x228u, ReadUInt32(memory, registerPacket + 4)); + Assert.Equal(markerPacket + (push ? 12UL : 8UL), registerPacket); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void Marker_WithoutCapacity_DoesNotReturnUnreservedStorage(bool push) + { + var (memory, ctx) = Setup(); + WriteUInt64(memory, CommandBufferAddress + 0x18, CursorAddress); + var before = Snapshot(memory); + ctx[CpuRegister.Rdi] = CommandBufferAddress; + ctx[CpuRegister.Rsi] = MarkerStringAddress; + if (push) AgcExports.DcbPushMarker(ctx); + else AgcExports.DcbPopMarker(ctx); + + Assert.Equal(0UL, ctx[CpuRegister.Rax]); + Assert.Equal(before, Snapshot(memory)); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task ForeignMarker_DoesNotOverwriteAnInlinePacketBeforeItsCursorIsPublished(bool push) + { + var (memory, _) = Setup(); + // Native guest threads each have a different tracker around the shared memory. + var writer = new CpuContext(new TrackedCpuMemory(memory), Generation.Gen5); + WriteShaderRegister(writer); // Establish the writer through an ordinary export. + var inlinePacket = ReadUInt64(memory, CommandBufferAddress + 0x10); + // Yotei writes SET_SH_REG in line before publishing its next cursor. The HLE + // allocation lock cannot protect these stores from another guest thread. + WriteUInt32(memory, inlinePacket, 0xC001_7600); + WriteUInt32(memory, inlinePacket + 4, 0x228); + WriteUInt32(memory, inlinePacket + 8, 0); + var before = Snapshot(memory); + var foreign = new CpuContext(new TrackedCpuMemory(memory), Generation.Gen5); + foreign[CpuRegister.Rdi] = CommandBufferAddress; + foreign[CpuRegister.Rsi] = MarkerStringAddress; + + await Task.Run(() => + { + if (push) AgcExports.DcbPushMarker(foreign); + else AgcExports.AcbPopMarker(foreign); + }); + + Assert.Equal(0UL, foreign[CpuRegister.Rax]); + Assert.Equal(before, Snapshot(memory)); + } + + private static void WriteShaderRegister(CpuContext ctx) + { + ctx[CpuRegister.Rdi] = CommandBufferAddress; + ctx[CpuRegister.Rsi] = 0x228; + ctx[CpuRegister.Rdx] = 0; + Assert.Equal(0, AgcExports.SetShaderRegisterDirect(ctx)); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void Marker_BufferFullCallback_UsesTheReplacementBufferAndPreservesReservedSpace(bool push) + { + var (memory, ctx) = Setup(); + const uint reserved = 7; + var packetDwords = push ? 3u : 2u; + WriteUInt64(memory, CommandBufferAddress + 0x18, CursorAddress + reserved * 4); + WriteUInt64(memory, CommandBufferAddress + 0x20, 0x1234); + WriteUInt64(memory, CommandBufferAddress + 0x28, 0x5678); + WriteUInt32(memory, CommandBufferAddress + 0x30, reserved); + var scheduler = DispatchProxy.Create(); + var proxy = (MarkerScheduler)(object)scheduler; + proxy.Callback = args => + { + Assert.Equal(0x1234UL, args[1]); + Assert.Equal(CommandBufferAddress, args[2]); + Assert.Equal((ulong)(packetDwords + reserved), args[3]); + Assert.Equal(0x5678UL, args[4]); + WriteUInt64(memory, CommandBufferAddress + 0x10, CursorAddress + 0x100); + WriteUInt64(memory, CommandBufferAddress + 0x18, CursorAddress + 0x100 + (packetDwords + reserved) * 4); + }; + var previous = GuestThreadExecution.Scheduler; + try + { + GuestThreadExecution.Scheduler = scheduler; + ctx[CpuRegister.Rdi] = CommandBufferAddress; + ctx[CpuRegister.Rsi] = MarkerStringAddress; + if (push) AgcExports.DcbPushMarker(ctx); + else AgcExports.DcbPopMarker(ctx); + Assert.Equal(1, proxy.CallCount); + Assert.Equal(CursorAddress + 0x100, ctx[CpuRegister.Rax]); + Assert.Equal(CursorAddress + 0x100 + packetDwords * 4, ReadUInt64(memory, CommandBufferAddress + 0x10)); + Assert.Equal(PoisonDword, ReadUInt32(memory, CursorAddress)); + } + finally + { + GuestThreadExecution.Scheduler = previous; + } + } + + [Fact] + public void OrdinaryExport_TransfersTheBufferToAnotherWriter() + { + var (memory, first) = Setup(); + WriteShaderRegister(first); + var second = new CpuContext(memory, Generation.Gen5); + WriteShaderRegister(second); + var cursor = ReadUInt64(memory, CommandBufferAddress + 0x10); + Assert.Equal(0, AgcExports.DcbPopMarker(second)); + Assert.Equal(cursor, second[CpuRegister.Rax]); + var before = Snapshot(memory); + Assert.Equal(0, AgcExports.DcbPopMarker(first)); + Assert.Equal(0UL, first[CpuRegister.Rax]); + Assert.Equal(before, Snapshot(memory)); + } + + [Fact] + public void WriterIdentity_UsesGuestHandlesAcrossContextsAndIsScopedToMemory() + { + var (memory, ctx) = Setup(); + var previous = GuestThreadExecution.EnterGuestThread(0x123); + try + { + WriteShaderRegister(ctx); + var resumed = new CpuContext(memory, Generation.Gen5); + resumed[CpuRegister.Rdi] = CommandBufferAddress; + var cursor = ReadUInt64(memory, CommandBufferAddress + 0x10); + AgcExports.AcbPopMarker(resumed); + Assert.Equal(cursor, resumed[CpuRegister.Rax]); + + GuestThreadExecution.EnterGuestThread(0x456); + var before = Snapshot(memory); + AgcExports.DcbPopMarker(ctx); + Assert.Equal(0UL, ctx[CpuRegister.Rax]); + Assert.Equal(before, Snapshot(memory)); + + // Reusing the same virtual address in a different game must not inherit an owner. + var (_, other) = Setup(); + other[CpuRegister.Rdi] = CommandBufferAddress; + AgcExports.DcbPopMarker(other); + Assert.Equal(CursorAddress, other[CpuRegister.Rax]); + } + finally + { + GuestThreadExecution.RestoreGuestThread(previous); + } + } + + public class MarkerScheduler : DispatchProxy + { + public Action? Callback { get; set; } + public int CallCount { get; private set; } + + protected override object? Invoke(MethodInfo? targetMethod, object?[]? args) + { + if (targetMethod?.Name != nameof(IGuestThreadScheduler.TryCallGuestFunction) || args?.Length != 10) + throw new NotSupportedException(targetMethod?.Name); + CallCount++; + Callback!(args); + args[8] = 0UL; + args[9] = null; + return true; + } + } + + [Fact] + public void Markers_RejectNullCommandBuffer() + { + var (_, ctx) = Setup(); + ctx[CpuRegister.Rdi] = 0; + ctx[CpuRegister.Rsi] = MarkerStringAddress; + + AgcExports.DcbPushMarker(ctx); + Assert.Equal(0UL, ctx[CpuRegister.Rax]); + AgcExports.DcbPopMarker(ctx); + Assert.Equal(0UL, ctx[CpuRegister.Rax]); + + } + + [Fact] + public void PushMarker_RejectsUnreadableString() + { + var (_, ctx) = Setup(); + ctx[CpuRegister.Rdi] = CommandBufferAddress; + ctx[CpuRegister.Rsi] = 0x10; // unmapped + + AgcExports.DcbPushMarker(ctx); + Assert.Equal(0UL, ctx[CpuRegister.Rax]); + + } + + [Fact] + public void Markers_WriteNopPackets() + { + var (memory, ctx) = Setup(); + ctx[CpuRegister.Rdi] = CommandBufferAddress; + ctx[CpuRegister.Rsi] = MarkerStringAddress; + + AgcExports.DcbPushMarker(ctx); + Assert.Equal(CursorAddress, ctx[CpuRegister.Rax]); + // "pass" + terminator = 2 payload dwords, plus the header. + Assert.Equal(0xC001_102Cu, ReadUInt32(memory, CursorAddress)); + Assert.Equal(CursorAddress + 12, ReadUInt64(memory, CommandBufferAddress + 0x10)); + + AgcExports.DcbPopMarker(ctx); + Assert.Equal(CursorAddress + 12, ctx[CpuRegister.Rax]); + Assert.Equal(0xC000_1030u, ReadUInt32(memory, CursorAddress + 12)); + Assert.Equal(CursorAddress + 20, ReadUInt64(memory, CommandBufferAddress + 0x10)); + + } + + private static (FakeCpuMemory Memory, CpuContext Ctx) Setup() + { + var memory = new FakeCpuMemory(BaseAddress, MemorySize); + var ctx = new CpuContext(memory, Generation.Gen5); + WriteUInt64(memory, CommandBufferAddress + 0x10, CursorAddress); + WriteUInt64(memory, CommandBufferAddress + 0x18, CursorAddress + 0x400); + memory.WriteCString(MarkerStringAddress, "pass"); + for (ulong offset = 0; offset < 0x40; offset += 4) + { + WriteUInt32(memory, CursorAddress + offset, PoisonDword); + } + + return (memory, ctx); + } + + private static byte[] Snapshot(FakeCpuMemory memory) + { + var buffer = new byte[MemorySize]; + Assert.True(memory.TryRead(BaseAddress, buffer)); + return buffer; + } + + private static uint ReadUInt32(FakeCpuMemory memory, ulong address) + { + Span buffer = stackalloc byte[4]; + Assert.True(memory.TryRead(address, buffer)); + return BinaryPrimitives.ReadUInt32LittleEndian(buffer); + } + + private static ulong ReadUInt64(FakeCpuMemory memory, ulong address) + { + Span buffer = stackalloc byte[8]; + Assert.True(memory.TryRead(address, buffer)); + return BinaryPrimitives.ReadUInt64LittleEndian(buffer); + } + + private static void WriteUInt64(FakeCpuMemory memory, ulong address, ulong value) + { + Span buffer = stackalloc byte[8]; + BinaryPrimitives.WriteUInt64LittleEndian(buffer, value); + Assert.True(memory.TryWrite(address, buffer)); + } + + private static void WriteUInt32(FakeCpuMemory memory, ulong address, uint value) + { + Span buffer = stackalloc byte[4]; + BinaryPrimitives.WriteUInt32LittleEndian(buffer, value); + Assert.True(memory.TryWrite(address, buffer)); + } +} diff --git a/tests/SharpEmu.Libs.Tests/Agc/Gen5MoveRelativeSpirvTests.cs b/tests/SharpEmu.Libs.Tests/Agc/Gen5MoveRelativeSpirvTests.cs index b99901e..1361ee6 100644 --- a/tests/SharpEmu.Libs.Tests/Agc/Gen5MoveRelativeSpirvTests.cs +++ b/tests/SharpEmu.Libs.Tests/Agc/Gen5MoveRelativeSpirvTests.cs @@ -10,8 +10,8 @@ using Xunit; namespace SharpEmu.Libs.Tests.Agc; -// Checks that relative register accesses resolve without a dynamically indexed register -// file, and rejects invalid relative sources. +// Checks that unknown relative indices preserve the full register bank, +// and rejects invalid relative sources. public sealed class Gen5MoveRelativeSpirvTests { private const ulong ShaderAddress = 0x1_0000_0000; @@ -33,7 +33,7 @@ public sealed class Gen5MoveRelativeSpirvTests // s_mov_b32 m0, s3 ; v_movrels_b32 v5, v3 -> v5 = vgpr[3 + m0] var spirv = Compile([SMovM0FromUserData, Vop1 | (5u << 17) | (0x43u << 9) | (256u + 3u)]); - AssertResolvesRelativeRegistersWithSelects(spirv, "V_MOVRELS_B32"); + AssertResolvesUnknownRelativeRegisters(spirv, "V_MOVRELS_B32"); } [Fact] @@ -42,7 +42,7 @@ public sealed class Gen5MoveRelativeSpirvTests // s_mov_b32 m0, s3 ; v_movreld_b32 v5, v3 -> vgpr[5 + m0] = v3 var spirv = Compile([SMovM0FromUserData, Vop1 | (5u << 17) | (0x42u << 9) | (256u + 3u)]); - AssertResolvesRelativeRegistersWithSelects(spirv, "V_MOVRELD_B32"); + AssertResolvesUnknownRelativeRegisters(spirv, "V_MOVRELD_B32"); } [Fact] @@ -51,7 +51,7 @@ public sealed class Gen5MoveRelativeSpirvTests // s_mov_b32 m0, s3 ; v_movrelsd_b32 v5, v3 -> vgpr[5 + m0] = vgpr[3 + m0] var spirv = Compile([SMovM0FromUserData, Vop1 | (5u << 17) | (0x44u << 9) | (256u + 3u)]); - AssertResolvesRelativeRegistersWithSelects(spirv, "V_MOVRELSD_B32"); + AssertResolvesUnknownRelativeRegisters(spirv, "V_MOVRELSD_B32"); } [Fact] @@ -61,7 +61,7 @@ public sealed class Gen5MoveRelativeSpirvTests // 10-bit halves (source index in [9:0], destination index in [25:16]). var spirv = Compile([SMovM0, Vop1 | (5u << 17) | (0x48u << 9) | (256u + 3u)]); - AssertResolvesRelativeRegistersWithSelects(spirv, "V_MOVRELSD_2_B32"); + AssertResolvesUnknownRelativeRegisters(spirv, "V_MOVRELSD_2_B32"); } [Theory] @@ -93,44 +93,11 @@ public sealed class Gen5MoveRelativeSpirvTests Assert.Contains("vector register", error, StringComparison.Ordinal); } - // A dynamically indexed private register array is lowered to thread memory on Metal, so no - // access chain may take a computed index; the relative access compares the computed register - // number with each candidate register (OpIEqual) and selects its value (OpSelect). - private static void AssertResolvesRelativeRegistersWithSelects(byte[] spirv, string opcode) + // Unknown indices require the full bank; bounded indices are tested separately. + private static void AssertResolvesUnknownRelativeRegisters(byte[] spirv, string opcode) { - var constants = new HashSet(); - var equalities = 0; - var selects = 0; - foreach (var (op, wordCount, offset) in EnumerateInstructions(spirv)) - { - // OpConstant = 43, OpConstantNull = 46: (opcode, resultType, resultId, ...). - if (op is 43 or 46 && wordCount >= 3) - { - constants.Add(ReadWord(spirv, offset + 8)); - } - - // OpIEqual = 170, OpSelect = 169. - equalities += op == 170 ? 1 : 0; - selects += op == 169 ? 1 : 0; - } - - foreach (var (op, wordCount, offset) in EnumerateInstructions(spirv)) - { - // OpAccessChain = 65: (opcode, resultType, resultId, base, index...). - if (op != 65) - { - continue; - } - - for (var index = 4; index < wordCount; index++) - { - Assert.True( - constants.Contains(ReadWord(spirv, offset + index * sizeof(uint))), - $"{opcode} must not index a register array with a computed index"); - } - } - - Assert.True(equalities > 1 && selects > 1, $"{opcode} must select the register its computed index names"); + Assert.True(HasDynamicVectorRegisterAccess(spirv), $"{opcode} must preserve every register its runtime index can name"); + SharpEmu.ShaderCompiler.Tests.Gen5LargeDispatcherValidationTests.ValidateWithSpirvToolsWhenAvailable(spirv); } // True when some access chain takes a computed index, i.e. registers are indexed at run time. diff --git a/tests/SharpEmu.Libs.Tests/Agc/Gen5ShaderAtomicDecodeTests.cs b/tests/SharpEmu.Libs.Tests/Agc/Gen5ShaderAtomicDecodeTests.cs index 9f8b06b..aba340f 100644 --- a/tests/SharpEmu.Libs.Tests/Agc/Gen5ShaderAtomicDecodeTests.cs +++ b/tests/SharpEmu.Libs.Tests/Agc/Gen5ShaderAtomicDecodeTests.cs @@ -152,6 +152,15 @@ public sealed class Gen5ShaderAtomicDecodeTests Assert.Equal("SSetprio", instruction.Opcode); } + [Fact] + public void SSleep_Decodes() + { + // S_SLEEP 2 (SOPP op 0x0E) + var instruction = DecodeSingle(0xBF8E0002); + + Assert.Equal("SSleep", instruction.Opcode); + } + [Fact] public void DsAddU32_HasAddressAndDataSourcesButNoDestination() { @@ -329,6 +338,36 @@ public sealed class Gen5ShaderAtomicDecodeTests Assert.Equal(new[] { Gen5Operand.Vector(1) }, instruction.Destinations); } + [Theory] + [InlineData(0x7E060903u, 0u, "VCvtF64I32")] + [InlineData(0x7E062D03u, 0u, "VCvtF64U32")] + [InlineData(0x7E065F03u, 0u, "VRcpF64")] + [InlineData(0x7E066103u, 0u, "VRsqF64")] + [InlineData(0x7E066903u, 0u, "VSqrtF64")] + [InlineData(0xD5650003u, 0x000E0B03u, "VMulF64")] + [InlineData(0xD54C0003u, 0x042E0B03u, "VFmaF64")] + [InlineData(0xD5840003u, 0x00000103u, "VCvtF64I32")] + [InlineData(0xD5960003u, 0x00000103u, "VCvtF64U32")] + [InlineData(0xD5AF0003u, 0x00000103u, "VRcpF64")] + [InlineData(0xD5B00003u, 0x00000103u, "VRsqF64")] + [InlineData(0xD5B40003u, 0x00000103u, "VSqrtF64")] + public void Binary64ResultWritesBothDestinationWords(uint first, uint second, string opcode) + { + var instruction = DecodeSingle(second == 0 ? [first] : [first, second]); + Assert.Equal(opcode, instruction.Opcode); + Assert.Equal(new[] { Gen5Operand.Vector(3), Gen5Operand.Vector(4) }, instruction.Destinations); + } + + [Theory] + [InlineData(0x7E061F03u, 0u)] + [InlineData(0xD58F0003u, 0x00000103u)] + public void Binary64ToBinary32WritesOnlyOneDestinationWord(uint first, uint second) + { + var instruction = DecodeSingle(second == 0 ? [first] : [first, second]); + Assert.Equal("VCvtF32F64", instruction.Opcode); + Assert.Equal(new[] { Gen5Operand.Vector(3) }, instruction.Destinations); + } + private static Gen5ShaderInstruction DecodeSingle(params uint[] words) { var memory = new FakeCpuMemory(ShaderAddress, 0x1000); diff --git a/tests/SharpEmu.Libs.Tests/Agc/Gen5TypedBufferLoadSpirvTests.cs b/tests/SharpEmu.Libs.Tests/Agc/Gen5TypedBufferLoadSpirvTests.cs index 6f3e073..476397f 100644 --- a/tests/SharpEmu.Libs.Tests/Agc/Gen5TypedBufferLoadSpirvTests.cs +++ b/tests/SharpEmu.Libs.Tests/Agc/Gen5TypedBufferLoadSpirvTests.cs @@ -24,14 +24,13 @@ public sealed class Gen5TypedBufferLoadSpirvTests } [Fact] - public void FormattedUntypedLoad_SpecializesOnTheDescriptorFormat() + public void FormattedUntypedLoad_DecodesItsSpecializedFormatWhenTranslated() { + // The specialized descriptor format is known here, so no run-time format table is emitted. var request = CreateCompileRequest("BufferLoadFormatXyzw", typed: false, typedFormat: 0, dwordCount: 4); Assert.Equal(77u, Assert.Single(request.Resources.Info.Buffers).DescriptorFormat); Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); - // The specialized format is a translation-time constant, so it decodes on the host and - // the module needs no run-time format lookup table at all. Assert.DoesNotContain(DescriptorFormatTableName, ModuleNames(shader.Spirv)); } @@ -45,12 +44,12 @@ public sealed class Gen5TypedBufferLoadSpirvTests } [Fact] - public void TypedD16Load_IsRejected() + public void TypedD16Load_Compiles() { + // Packed D16 loads are translated; PackedD16BufferTests checks their results on the device. var request = CreateCompileRequest("TBufferLoadFormatD16Xy", typed: true, typedFormat: 64, dwordCount: 1); - Assert.False(Gen5SpirvTranslator.TryCompileProgram(request, out _, out var error)); - Assert.Contains("TBufferLoadFormatD16Xy", error, StringComparison.Ordinal); + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out _, out var error), error); } [Fact] @@ -168,4 +167,16 @@ public sealed class Gen5TypedBufferLoadSpirvTests return names; } + + [Fact] + public void FormattedUntypedLoad_SpecializesOnTheDescriptorFormat() + { + var request = CreateCompileRequest("BufferLoadFormatXyzw", typed: false, typedFormat: 0, dwordCount: 4); + Assert.Equal(77u, Assert.Single(request.Resources.Info.Buffers).DescriptorFormat); + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + + // The specialized format is a translation-time constant, so it decodes on the host and + // the module needs no run-time format lookup table at all. + Assert.DoesNotContain(DescriptorFormatTableName, ModuleNames(shader.Spirv)); + } } diff --git a/tests/SharpEmu.Libs.Tests/Audio/AcmExportsTests.cs b/tests/SharpEmu.Libs.Tests/Audio/AcmExportsTests.cs index 12e57c0..47616ba 100644 --- a/tests/SharpEmu.Libs.Tests/Audio/AcmExportsTests.cs +++ b/tests/SharpEmu.Libs.Tests/Audio/AcmExportsTests.cs @@ -102,6 +102,34 @@ public sealed class AcmExportsTests : IDisposable } } + [Fact] + public void ConvReverbSharedInput_ClearsOutputBuffers() + { + const ulong stackAddress = MemoryBase + 0x600; + const ulong descriptorListAddress = MemoryBase + 0x680; + const ulong descriptorAddress = MemoryBase + 0x700; + const ulong bufferListAddress = MemoryBase + 0x740; + const ulong outputAddress = MemoryBase + 0x780; + + Span output = stackalloc byte[16]; + output.Fill(0x7F); + Assert.True(_memory.TryWrite(outputAddress, output)); + WriteUInt64(bufferListAddress, outputAddress); + WriteUInt64(descriptorListAddress, descriptorAddress); + + Span descriptor = stackalloc byte[24]; + BinaryPrimitives.WriteUInt32LittleEndian(descriptor, 4); + BinaryPrimitives.WriteUInt32LittleEndian(descriptor[4..], 1); + BinaryPrimitives.WriteUInt64LittleEndian(descriptor[16..], bufferListAddress); + Assert.True(_memory.TryWrite(descriptorAddress, descriptor)); + + _ctx[CpuRegister.Rsp] = stackAddress; + WriteUInt64(stackAddress + 8, descriptorListAddress); + _ctx[CpuRegister.Rcx] = 1; + Assert.Equal(0, AcmExports.AcmConvReverbSharedInput(_ctx)); + Assert.All(ReadBytes(outputAddress, 16), value => Assert.Equal(0, value)); + } + public void Dispose() { AcmExports.ResetForTests(); diff --git a/tests/SharpEmu.Libs.Tests/Audio/AjmExportsTests.cs b/tests/SharpEmu.Libs.Tests/Audio/AjmExportsTests.cs index 49532b3..2f9f815 100644 --- a/tests/SharpEmu.Libs.Tests/Audio/AjmExportsTests.cs +++ b/tests/SharpEmu.Libs.Tests/Audio/AjmExportsTests.cs @@ -2,6 +2,7 @@ // SPDX-License-Identifier: GPL-2.0-or-later using System.Buffers.Binary; +using System.Reflection; using SharpEmu.HLE; using SharpEmu.Libs.Audio; using Xunit; @@ -28,6 +29,8 @@ public sealed class AjmExportsTests : IDisposable private const ulong BatchInfoAddress = MemoryBase + 0x300; private const ulong StatisticsAddress = MemoryBase + 0x400; private const ulong BatchBufferAddress = MemoryBase + 0x500; + private const ulong ConfigAddress = MemoryBase + 0xB00; + private const ulong ConfigInfoAddress = MemoryBase + 0xB10; private readonly FakeCpuMemory _memory = new(MemoryBase, 0x1000); private readonly CpuContext _ctx; @@ -198,6 +201,105 @@ public sealed class AjmExportsTests : IDisposable Assert.Equal(0x4001u, ReadUInt32(InstanceAddress)); } + [Fact] + public void ParseConfigData_LayoutSevenIsReportedAsStereo() + { + WriteBytes(ConfigAddress, [0xFE, 0x4E, 0x00, 0x40]); + + _ctx[CpuRegister.Rdi] = ConfigAddress; + _ctx[CpuRegister.Rsi] = ConfigInfoAddress; + Assert.Equal(0, AjmExports.AjmDecAt9ParseConfigData(_ctx)); + + Assert.Equal(3u, ReadUInt32(ConfigInfoAddress)); + Assert.Equal(1u, ReadUInt32(ConfigInfoAddress + 4)); + Assert.Equal(2u, ReadUInt32(ConfigInfoAddress + 8)); + Assert.Equal(128u, ReadUInt32(ConfigInfoAddress + 12)); + Assert.Equal(3u, ReadUInt32(ConfigInfoAddress + 16)); + } + + [Fact] + public void ParseConfigData_Ps5MultichannelLayoutReportsFixedSuperframe() + { + WriteBytes(ConfigAddress, [0x30, 0x62, 0xC0, 0x40]); + + _ctx[CpuRegister.Rdi] = ConfigAddress; + _ctx[CpuRegister.Rsi] = ConfigInfoAddress; + Assert.Equal(0, AjmExports.AjmDecAt9ParseConfigData(_ctx)); + + Assert.Equal(204u, ReadUInt32(ConfigInfoAddress)); + Assert.Equal(1u, ReadUInt32(ConfigInfoAddress + 4)); + Assert.Equal(12u, ReadUInt32(ConfigInfoAddress + 8)); + Assert.Equal(256u, ReadUInt32(ConfigInfoAddress + 12)); + Assert.Equal(17u, ReadUInt32(ConfigInfoAddress + 16)); + } + + [Fact] + public void ParseConfigData_UsesSuperframeIndexInMonoDecoderConfig() + { + var decoder = new Atrac9DecodeState(); + Assert.True(decoder.TryInitialize([0x30, 0x62, 0xC0, 0x42])); + + var configField = typeof(Atrac9DecodeState).GetField( + "_extendedDecoderConfig", + BindingFlags.Instance | BindingFlags.NonPublic); + var monoConfig = Assert.IsType(configField?.GetValue(decoder)); + + // frameBytes=17 and sfIndex=2: the low config field is 16 << 5 | 2 << 3. + Assert.Equal(new byte[] { 0xFE, 0x60, 0x02, 0x10 }, monoConfig); + } + + [Fact] + public void ExtendedBlocks_AdvanceByConsumedBytesAndSkipPadding() + { + // Two channels, two frames: the consumed block lengths differ and the + // second frame has low-bit padding before its second channel block. + ReadOnlySpan compressed = + [ + 0x80, 0x01, // frame 0, channel 0: used = 2 + 0x80, 0x82, // frame 0, channel 1: used = 2 + 0x80, // frame 1, channel 0: used = 1 + 0x03, 0x04, 0x80, // padding before channel 1 + 0x80, 0x05, // frame 1, channel 1: used = 2 + ]; + var usedBytes = new[] { 2, 2, 1, 2 }; + var position = 0; + var usedIndex = 0; + + for (var frame = 0; frame < 2; frame++) + { + for (var channel = 0; channel < 2; channel++) + { + Assert.True(Atrac9DecodeState.TryBeginExtendedBlock(compressed, frame, ref position)); + position += usedBytes[usedIndex++]; + } + } + + Assert.Equal(9, position); + } + + [Fact] + public void UnsupportedStandardLayoutProducesSilentSuperframeWithoutError() + { + // Layout 6 is outside LibAtrac9's standard channel table. It must not + // be handed to the decoder, but the header still describes a complete + // stereo-duration superframe so the caller can keep its timeline. + var decoder = new Atrac9DecodeState(); + Assert.True(decoder.TryInitialize([0xFE, 0x4C, 0x00, 0x40])); + + var output = new byte[128 * 2 * sizeof(short)]; + var result = decoder.Decode( + [0x00, 0x00, 0x00], + output, + Atrac9PcmEncoding.Signed16, + requestedChannels: 2, + multipleFrames: false); + + Assert.Equal(0, result.Status); + Assert.Equal(output.Length, result.OutputWritten); + Assert.Equal(128u, result.TotalDecodedSamples); + Assert.All(output, value => Assert.Equal(0, value)); + } + /// /// sceAjmBatchJobRunSplit gathers arrays of AjmBuffer descriptors rather than /// a single pointer/size pair, and its sideband layout is positional: result, @@ -429,6 +531,88 @@ public sealed class AjmExportsTests : IDisposable return AjmExports.AjmModuleRegister(_ctx); } + [Fact] + public void BatchJobDecodeSplit_InvalidInstanceWritesResultThroughSeventhArgument() + { + const ulong resultAddress = MemoryBase + 0xB00; + const ulong outputAddress = MemoryBase + 0xC00; + InitializeBatch(BatchBufferAddress, 0x100, BatchInfoAddress); + WriteUInt64(outputAddress, 0x1122334455667788); + WriteUInt64(resultAddress + 32, 0x8877665544332211); + WriteStackArgs(resultAddress, 0, 0); + _ctx[CpuRegister.Rdi] = BatchInfoAddress; + _ctx[CpuRegister.Rsi] = uint.MaxValue; + _ctx[CpuRegister.Rdx] = MemoryBase + 0xD00; + _ctx[CpuRegister.Rcx] = 1; + _ctx[CpuRegister.R8] = outputAddress; + _ctx[CpuRegister.R9] = 1; + + Assert.Equal(0, AjmExports.AjmBatchJobDecodeSplit(_ctx)); + Assert.NotEqual(0u, ReadUInt32(resultAddress)); + Assert.Equal(0u, ReadUInt32(resultAddress + 8)); + Assert.Equal(0u, ReadUInt32(resultAddress + 12)); + Assert.Equal(0UL, ReadUInt64(resultAddress + 16)); + Assert.Equal(0u, ReadUInt32(resultAddress + 24)); + Assert.Equal(0x1122334455667788UL, ReadUInt64(outputAddress)); + Assert.Equal(0x8877665544332211UL, ReadUInt64(resultAddress + 32)); + } + + [Fact] + public void BatchJobDecodeSplit_GathersAFrameAndScattersPcmWithoutOverwritingBoundaries() + { + const ulong configAddress = MemoryBase + 0x600; + const ulong inputDescriptors = MemoryBase + 0x620; + const ulong outputDescriptors = MemoryBase + 0x660; + const ulong inputData = MemoryBase + 0x700; + const ulong outputData = MemoryBase + 0x800; + const ulong resultAddress = MemoryBase + 0xB00; + var contextId = Initialize(); + Assert.Equal(0, RegisterCodec(contextId, 1)); + Assert.Equal(0, CreateInstance(contextId, 1, 0x401, InstanceAddress)); + var instanceId = ReadUInt32(InstanceAddress); + InitializeBatch(BatchBufferAddress, 0x100, BatchInfoAddress); + Assert.True(_memory.TryWrite(configAddress, new byte[] { 0xFE, 0x70, 0x17, 0xE0 })); + _ctx[CpuRegister.Rdi] = BatchInfoAddress; + _ctx[CpuRegister.Rsi] = instanceId; + _ctx[CpuRegister.Rdx] = configAddress; + _ctx[CpuRegister.Rcx] = 4; + _ctx[CpuRegister.R8] = resultAddress; + Assert.Equal(0, AjmExports.AjmBatchJobInitialize(_ctx)); + Assert.Equal(0u, ReadUInt32(resultAddress)); + + var input = new byte[192]; + new byte[] { 0x00, 0x84, 0x00, 0x40 }.CopyTo(input, 0); + Assert.True(_memory.TryWrite(inputData, input)); + WriteUInt64(inputDescriptors, inputData); + WriteUInt64(inputDescriptors + 8, 2); + WriteUInt64(inputDescriptors + 16, inputData + 2); + WriteUInt64(inputDescriptors + 24, 190); + WriteUInt64(outputDescriptors, outputData); + WriteUInt64(outputDescriptors + 8, 128); + WriteUInt64(outputDescriptors + 16, outputData + 144); + WriteUInt64(outputDescriptors + 24, 384); + var sentinel = new byte[544]; + Array.Fill(sentinel, (byte)0xCC); + Assert.True(_memory.TryWrite(outputData, sentinel)); + _ctx[CpuRegister.Rsp] = MemoryBase + 0xF00; + WriteUInt64(MemoryBase + 0xF08, resultAddress); + _ctx[CpuRegister.Rdi] = BatchInfoAddress; + _ctx[CpuRegister.Rsi] = instanceId; + _ctx[CpuRegister.Rdx] = inputDescriptors; + _ctx[CpuRegister.Rcx] = 2; + _ctx[CpuRegister.R8] = outputDescriptors; + _ctx[CpuRegister.R9] = 2; + Assert.Equal(0, AjmExports.AjmBatchJobDecodeSplit(_ctx)); + Assert.Equal(0u, ReadUInt32(resultAddress)); + Assert.Equal(192u, ReadUInt32(resultAddress + 8)); + Assert.Equal(512u, ReadUInt32(resultAddress + 12)); + Assert.Equal(256UL, ReadUInt64(resultAddress + 16)); + Assert.All(ReadBytes(outputData, 128), value => Assert.Equal(0, value)); + Assert.All(ReadBytes(outputData + 128, 16), value => Assert.Equal(0xCC, value)); + Assert.All(ReadBytes(outputData + 144, 384), value => Assert.Equal(0, value)); + Assert.All(ReadBytes(outputData + 528, 16), value => Assert.Equal(0xCC, value)); + } + private int UnregisterCodec(uint contextId, uint codecType) { _ctx[CpuRegister.Rdi] = contextId; @@ -516,6 +700,8 @@ public sealed class AjmExportsTests : IDisposable return value; } + private void WriteBytes(ulong address, ReadOnlySpan value) => Assert.True(_memory.TryWrite(address, value)); + private void WriteUInt32(ulong address, uint value) { Span bytes = stackalloc byte[sizeof(uint)]; diff --git a/tests/SharpEmu.Libs.Tests/Audio/AudioOut2PortGetStateExportsTests.cs b/tests/SharpEmu.Libs.Tests/Audio/AudioOut2PortGetStateExportsTests.cs index d3ab7b5..328273e 100644 --- a/tests/SharpEmu.Libs.Tests/Audio/AudioOut2PortGetStateExportsTests.cs +++ b/tests/SharpEmu.Libs.Tests/Audio/AudioOut2PortGetStateExportsTests.cs @@ -19,6 +19,78 @@ public sealed class AudioOut2PortGetStateExportsTests return new CpuContext(memory, Generation.Gen5); } + [Theory] + [InlineData(0x100u, 1)] + [InlineData(0x201u, 2)] + [InlineData(0x880u, 8)] + public void PortGetState_ExposesChannelCountInTheGuestSdkField(uint format, byte channels) + { + var ctx = CreateContext(out var memory); + Span parameters = stackalloc byte[0x40]; + parameters.Clear(); + BinaryPrimitives.WriteUInt32LittleEndian(parameters[4..], format); + BinaryPrimitives.WriteUInt32LittleEndian(parameters[8..], 48000); + Assert.True(memory.TryWrite(MemoryBase, parameters)); + ctx[CpuRegister.Rdi] = 0; + ctx[CpuRegister.Rsi] = MemoryBase; + ctx[CpuRegister.Rdx] = MemoryBase + 0x80; + Assert.Equal(0, AudioOut2Exports.AudioOut2PortCreate(ctx)); + Span handleBytes = stackalloc byte[8]; + Assert.True(memory.TryRead(MemoryBase + 0x80, handleBytes)); + var handle = BinaryPrimitives.ReadUInt64LittleEndian(handleBytes); + try + { + ctx[CpuRegister.Rdi] = handle; + ctx[CpuRegister.Rsi] = StateAddress; + Assert.Equal(0, AudioOut2Exports.AudioOut2PortGetState(ctx)); + Span state = stackalloc byte[0x40]; + Assert.True(memory.TryRead(StateAddress, state)); + // Demon's Souls selects its speaker layout from +2. Yotei also reads + // +2 and waits until it is nonzero, which the channel count satisfies. + Assert.Equal(channels, state[2]); + Assert.NotEqual(0, state[2]); + Assert.Equal(0, state[3]); + } + finally + { + ctx[CpuRegister.Rdi] = handle; + Assert.Equal(0, AudioOut2Exports.AudioOut2PortDestroy(ctx)); + } + } + + [Fact] + public void ContextPush_NonblockingWithoutPcmDoesNotPaceSilentGrains() + { + var ctx = CreateContext(out var memory); + Span parameters = stackalloc byte[0x40]; + parameters.Clear(); + BinaryPrimitives.WriteUInt32LittleEndian(parameters[0x0C..], 4); + BinaryPrimitives.WriteUInt32LittleEndian(parameters[0x10..], 0x4000); + Assert.True(memory.TryWrite(MemoryBase, parameters)); + ctx[CpuRegister.Rdi] = MemoryBase; + ctx[CpuRegister.Rsi] = MemoryBase + 0x400; + ctx[CpuRegister.Rdx] = 0x100; + ctx[CpuRegister.Rcx] = StateAddress; + Assert.Equal(0, AudioOut2Exports.AudioOut2ContextCreate(ctx)); + Span handleBytes = stackalloc byte[8]; + Assert.True(memory.TryRead(StateAddress, handleBytes)); + var handle = BinaryPrimitives.ReadUInt64LittleEndian(handleBytes); + try + { + ctx[CpuRegister.Rdi] = handle; + ctx[CpuRegister.Rsi] = 0; + var elapsed = System.Diagnostics.Stopwatch.StartNew(); + for (var i = 0; i < 4; i++) + Assert.Equal(0, AudioOut2Exports.AudioOut2ContextPush(ctx)); + // Incorrectly pacing these grains adds over a second of sleep. + Assert.True(elapsed.Elapsed < TimeSpan.FromMilliseconds(500)); + } + finally + { + ctx[CpuRegister.Rdi] = handle; + Assert.Equal(0, AudioOut2Exports.AudioOut2ContextDestroy(ctx)); + } + } [Fact] public void PortGetState_WritesExactlySizeofPortStateIgnoringPollutedR9() { @@ -39,7 +111,7 @@ public sealed class AudioOut2PortGetStateExportsTests Assert.Equal(0, result); Span state = stackalloc byte[0x100]; Assert.True(memory.TryRead(StateAddress, state)); - // SceAudioOut2PortState: output@0, numChannels@2, volume@4. + // SceAudioOut2PortState: output@0, numChannels@2, padding@3, volume@4. Assert.Equal(1, BinaryPrimitives.ReadUInt16LittleEndian(state)); Assert.Equal(2, state[2]); Assert.Equal(0, state[3]); @@ -48,6 +120,7 @@ public sealed class AudioOut2PortGetStateExportsTests Assert.Equal(0u, BinaryPrimitives.ReadUInt32LittleEndian(state[8..])); Assert.Equal(0u, BinaryPrimitives.ReadUInt32LittleEndian(state[0x0C..])); Assert.All(state[0x10..0x40].ToArray(), value => Assert.Equal(0, value)); + // Bytes past sizeof(SceAudioOut2PortState), 0x40, must remain untouched. Assert.Equal(0xAB, state[0x40]); Assert.Equal(0xAB, state[0x7F]); } diff --git a/tests/SharpEmu.Libs.Tests/Audio/AudioOut2QueueLevelExportsTests.cs b/tests/SharpEmu.Libs.Tests/Audio/AudioOut2QueueLevelExportsTests.cs new file mode 100644 index 0000000..7b0150c --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Audio/AudioOut2QueueLevelExportsTests.cs @@ -0,0 +1,100 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Buffers.Binary; +using SharpEmu.HLE; +using SharpEmu.Libs.Audio; +using Xunit; + +namespace SharpEmu.Libs.Tests.Audio; + +[CollectionDefinition("AudioOut2QueueState", DisableParallelization = true)] +public sealed class AudioOut2QueueStateCollection +{ + public const string Name = "AudioOut2QueueState"; +} + +[Collection(AudioOut2QueueStateCollection.Name)] +public sealed class AudioOut2QueueLevelExportsTests +{ + private const ulong MemoryBase = 0x1_3000_0000; + private const ulong ParamAddress = MemoryBase + 0x100; + private const ulong ContextMemoryAddress = MemoryBase + 0x200; + private const ulong ContextOutAddress = MemoryBase + 0x300; + private const ulong QueuedAddress = MemoryBase + 0x400; + private const ulong FreeAddress = MemoryBase + 0x404; + + [Fact] + public void ContextGetQueueLevelReportsFreeGrainsAsSecondValue() + { + var memory = new FakeCpuMemory(MemoryBase, 0x1000); + var ctx = new CpuContext(memory, Generation.Gen5); + Span parameters = stackalloc byte[0x40]; + BinaryPrimitives.WriteUInt32LittleEndian(parameters[0x0C..], 4); + BinaryPrimitives.WriteUInt32LittleEndian(parameters[0x10..], 256); + Assert.True(memory.TryWrite(ParamAddress, parameters)); + + ctx[CpuRegister.Rdi] = ParamAddress; + ctx[CpuRegister.Rsi] = ContextMemoryAddress; + ctx[CpuRegister.Rdx] = 0x4000; + ctx[CpuRegister.Rcx] = ContextOutAddress; + Assert.Equal(0, AudioOut2Exports.AudioOut2ContextCreate(ctx)); + var handle = ReadUInt64(memory, ContextOutAddress); + + ctx[CpuRegister.Rdi] = handle; + ctx[CpuRegister.Rsi] = QueuedAddress; + ctx[CpuRegister.Rdx] = FreeAddress; + Assert.Equal(0, AudioOut2Exports.AudioOut2ContextGetQueueLevel(ctx)); + Assert.Equal(0u, ReadUInt32(memory, QueuedAddress)); + Assert.Equal(4u, ReadUInt32(memory, FreeAddress)); + + ctx[CpuRegister.Rdi] = handle; + Assert.Equal(0, AudioOut2Exports.AudioOut2ContextDestroy(ctx)); + } + + [Fact] + public void ContextGetQueueLevelUsesSoftwarePacingWithoutBackend() + { + var memory = new FakeCpuMemory(MemoryBase, 0x1000); + var ctx = new CpuContext(memory, Generation.Gen5); + Span parameters = stackalloc byte[0x40]; + BinaryPrimitives.WriteUInt32LittleEndian(parameters[0x0C..], 4); + BinaryPrimitives.WriteUInt32LittleEndian(parameters[0x10..], 4096); + Assert.True(memory.TryWrite(ParamAddress, parameters)); + + ctx[CpuRegister.Rdi] = ParamAddress; + ctx[CpuRegister.Rsi] = ContextMemoryAddress; + ctx[CpuRegister.Rdx] = 0x4000; + ctx[CpuRegister.Rcx] = ContextOutAddress; + Assert.Equal(0, AudioOut2Exports.AudioOut2ContextCreate(ctx)); + var handle = ReadUInt64(memory, ContextOutAddress); + + // No port is attached, so ContextPush cannot bind a host backend and + // must account for the grain through the software pacing clock. + ctx[CpuRegister.Rdi] = handle; + ctx[CpuRegister.Rsi] = 0; + Assert.Equal(0, AudioOut2Exports.AudioOut2ContextPush(ctx)); + + ctx[CpuRegister.Rdi] = handle; + ctx[CpuRegister.Rsi] = QueuedAddress; + ctx[CpuRegister.Rdx] = 0; + Assert.Equal(0, AudioOut2Exports.AudioOut2ContextGetQueueLevel(ctx)); + Assert.InRange(ReadUInt32(memory, QueuedAddress), 1u, 4u); + + Assert.Equal(0, AudioOut2Exports.AudioOut2ContextDestroy(ctx)); + } + + private static uint ReadUInt32(FakeCpuMemory memory, ulong address) + { + Span value = stackalloc byte[sizeof(uint)]; + Assert.True(memory.TryRead(address, value)); + return BinaryPrimitives.ReadUInt32LittleEndian(value); + } + + private static ulong ReadUInt64(FakeCpuMemory memory, ulong address) + { + Span value = stackalloc byte[sizeof(ulong)]; + Assert.True(memory.TryRead(address, value)); + return BinaryPrimitives.ReadUInt64LittleEndian(value); + } +} diff --git a/tests/SharpEmu.Libs.Tests/AvPlayer/AvPlayerAbiTests.cs b/tests/SharpEmu.Libs.Tests/AvPlayer/AvPlayerAbiTests.cs index 0643e54..8781ee9 100644 --- a/tests/SharpEmu.Libs.Tests/AvPlayer/AvPlayerAbiTests.cs +++ b/tests/SharpEmu.Libs.Tests/AvPlayer/AvPlayerAbiTests.cs @@ -334,3 +334,47 @@ public sealed class AvPlayerAbiTests hasPresentation)); } } + +public sealed class Videodec2PictureInfoAbiTests +{ + [Fact] + public void DimensionsPreservePitchAndAdjacentFields() + { + const ulong address = 0x100000000; + var memory = new FakeCpuMemory(address, 0x100); + var context = new CpuContext(memory, Generation.Gen5); + var expected = Enumerable.Repeat((byte)0xab, 0x38).ToArray(); + Assert.True(memory.TryWrite(address, expected)); + Assert.True(SharpEmu.Libs.Codec.Videodec2Exports.TryWritePictureDimensions(context, address, 1920, 1080)); + BinaryPrimitives.WriteUInt32LittleEndian(expected.AsSpan(0x10), 1920); + BinaryPrimitives.WriteUInt32LittleEndian(expected.AsSpan(0x18), 1080); + var actual = new byte[expected.Length]; + Assert.True(memory.TryRead(address, actual)); + Assert.Equal(expected, actual); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void NoPicturePreservesSizeAndClearsTheReadyField(bool flush) + { + const ulong address = 0x100000000; + var memory = new FakeCpuMemory(address, 0x100); + var context = new CpuContext(memory, Generation.Gen5); + var initial = Enumerable.Repeat((byte)0xab, 0x48).ToArray(); + BinaryPrimitives.WriteUInt64LittleEndian(initial, 0x48); + Assert.True(memory.TryWrite(address, initial)); + context[CpuRegister.Rdi] = ulong.MaxValue; + context[flush ? CpuRegister.Rdx : CpuRegister.Rcx] = address; + // Both calls receive the frame buffer, whose accepted flag they clear. + context[flush ? CpuRegister.Rsi : CpuRegister.Rdx] = address + 0x60; + Assert.Equal(0, flush + ? SharpEmu.Libs.Codec.Videodec2Exports.Videodec2Flush(context) + : SharpEmu.Libs.Codec.Videodec2Exports.Videodec2Decode(context)); + var actual = new byte[initial.Length]; + Assert.True(memory.TryRead(address, actual)); + // No picture clears the ready flag, the error flag and the picture count. + initial.AsSpan(8, 4).Clear(); + Assert.Equal(initial, actual); + } +} diff --git a/tests/SharpEmu.Libs.Tests/Cli/RosettaGcBudgetTests.cs b/tests/SharpEmu.Libs.Tests/Cli/RosettaGcBudgetTests.cs new file mode 100644 index 0000000..52537c1 --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Cli/RosettaGcBudgetTests.cs @@ -0,0 +1,26 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.CLI; +using Xunit; + +namespace SharpEmu.Libs.Tests.Cli; + +public sealed class RosettaGcBudgetTests +{ + [Fact] + public void DotnetRestartKeepsTheManagedEntryAssemblyAndGuestArguments() + { + Assert.Equal(new[] { "/runtime/dotnet", "/emulator/SharpEmu.CLI.dll", "--hdr=off", "/games/My Game/eboot.bin" }, + RosettaGcBudget.BuildArguments("/runtime/dotnet", "/emulator/SharpEmu.CLI.dll", + ["--hdr=off", "/games/My Game/eboot.bin"])); + } + + [Fact] + public void AppHostRestartDoesNotPassTheManagedAssemblyAsAGuestArgument() + { + Assert.Equal(new[] { "/emulator/SharpEmu.CLI", "--hdr=off", "/games/My Game/eboot.bin" }, + RosettaGcBudget.BuildArguments("/emulator/SharpEmu.CLI", "/emulator/SharpEmu.CLI.dll", + ["--hdr=off", "/games/My Game/eboot.bin"])); + } +} diff --git a/tests/SharpEmu.Libs.Tests/Cpu/GuestFastPathStubTests.cs b/tests/SharpEmu.Libs.Tests/Cpu/GuestFastPathStubTests.cs index 4b84faf..9c3e138 100644 --- a/tests/SharpEmu.Libs.Tests/Cpu/GuestFastPathStubTests.cs +++ b/tests/SharpEmu.Libs.Tests/Cpu/GuestFastPathStubTests.cs @@ -297,6 +297,30 @@ public sealed unsafe class GuestFastPathStubTests Assert.Equal(original, buffer); } + [Fact] + public void MemoryCopyStub_LeavesAPageSizedCopyToTheManagedExport() + { + if (!CanRunStubs()) + { + return; + } + + using var stub = new MemoryCopyStub(); + var minimum = DirectExecutionBackend.ManagedCopyMinimumBytes; + var source = Enumerable.Range(0, minimum).Select(value => (byte)(value * 3)).ToArray(); + var destination = new byte[minimum]; + fixed (byte* from = source) + fixed (byte* to = destination) + { + // The managed export marks the whole destination written before it copies, instead of + // taking a write-tracking fault per watched page. + Assert.Equal(MemoryCopySlowPathMarker, stub.Call(to, from, (nuint)minimum)); + Assert.Equal((ulong)to, stub.Call(to, from, (nuint)minimum - 1)); + } + + Assert.Equal(source[..^1], destination[..^1]); + } + /// The memcpy stub called with the platform (SysV) convention the guest uses. private sealed class MemoryCopyStub : IDisposable { diff --git a/tests/SharpEmu.Libs.Tests/Cpu/GuestThreadRegionAllocationTests.cs b/tests/SharpEmu.Libs.Tests/Cpu/GuestThreadRegionAllocationTests.cs new file mode 100644 index 0000000..9bd49d9 --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Cpu/GuestThreadRegionAllocationTests.cs @@ -0,0 +1,72 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.Core.Cpu.Native; +using SharpEmu.Core.Loader; +using SharpEmu.Core.Memory; +using Xunit; + +namespace SharpEmu.Libs.Tests.Cpu; + +public sealed class GuestThreadRegionAllocationTests +{ + // Two host threads that each claim a nested-callback stack at the same time must get + // different regions. Map reuses a region that already exists, so a claim that checks + // for a free slot and maps it in two steps hands both threads the same stack: Ghost of + // Yotei's render threads then returned into each other's host stacks. + [Fact] + public void ConcurrentClaims_GetDistinctRegions() + { + var memory = new InterleavingMemory(); + var bases = new ulong[2]; + var claims = Enumerable.Range(0, 2) + .Select(index => new Thread(() => + { + Assert.True(DirectExecutionBackend.TryMapGuestThreadRegion( + memory, 0x7FFF_E000_0000UL, 0x20_0000UL, + ProgramHeaderFlags.Read | ProgramHeaderFlags.Write, + out bases[index], out var error), error); + })) + .ToList(); + claims.ForEach(thread => thread.Start()); + claims.ForEach(thread => thread.Join()); + + Assert.NotEqual(bases[0], bases[1]); + } + + // Region bookkeeping only. SnapshotRegions waits briefly for the other claimer, so two + // claims that are not serialized both look for a free slot before either maps one. + private sealed class InterleavingMemory : IVirtualMemory + { + private readonly List _regions = []; + private readonly Barrier _bothLooking = new(2); + + public void Clear() => _regions.Clear(); + + public bool IsBackedView(ulong address) => false; + + public void Map(ulong virtualAddress, ulong memorySize, ulong fileOffset, ReadOnlySpan fileData, ProgramHeaderFlags protection) + { + lock (_regions) + { + if (!_regions.Any(region => region.VirtualAddress == virtualAddress)) + { + _regions.Add(new VirtualMemoryRegion(virtualAddress, memorySize, fileOffset, 0, protection)); + } + } + } + + public IReadOnlyList SnapshotRegions() + { + _bothLooking.SignalAndWait(TimeSpan.FromMilliseconds(500)); + lock (_regions) + { + return [.. _regions]; + } + } + + public bool TryRead(ulong virtualAddress, Span destination) => false; + + public bool TryWrite(ulong virtualAddress, ReadOnlySpan source) => false; + } +} diff --git a/tests/SharpEmu.Libs.Tests/Cpu/ImportLoopGuardBoundaryTests.cs b/tests/SharpEmu.Libs.Tests/Cpu/ImportLoopGuardBoundaryTests.cs index 4162217..035a0d5 100644 --- a/tests/SharpEmu.Libs.Tests/Cpu/ImportLoopGuardBoundaryTests.cs +++ b/tests/SharpEmu.Libs.Tests/Cpu/ImportLoopGuardBoundaryTests.cs @@ -13,6 +13,28 @@ namespace SharpEmu.Libs.Tests.Cpu; public sealed class ImportLoopGuardBoundaryTests { + [Fact] + public void UnresolvedWarningsSampleEachNidIndependentlyAcrossConcurrentCalls() + { + var backend = (DirectExecutionBackend)RuntimeHelpers.GetUninitializedObject(typeof(DirectExecutionBackend)); + const BindingFlags flags = BindingFlags.Instance | BindingFlags.NonPublic; + typeof(DirectExecutionBackend).GetField("_importResultLogSampleGate", flags)!.SetValue(backend, new object()); + typeof(DirectExecutionBackend).GetField("_unresolvedImportLogSamples", flags)!.SetValue( + backend, new Dictionary(StringComparer.Ordinal)); + var method = typeof(DirectExecutionBackend).GetMethod("ShouldLogUnresolvedImport", flags)!; + var emitted = new System.Collections.Concurrent.ConcurrentBag(); + Parallel.For(0, 20001, _ => + { + object?[] arguments = ["missing-export", 0L]; + if (Assert.IsType(method.Invoke(backend, arguments))) + emitted.Add(Assert.IsType(arguments[1])); + }); + Assert.Equal(new long[] { 1, 2, 3, 4, 5, 6, 7, 8, 10000, 20000 }, emitted.Order()); + object?[] otherArguments = ["another-export", 0L]; + Assert.True(Assert.IsType(method.Invoke(backend, otherArguments))); + Assert.Equal(1L, Assert.IsType(otherArguments[1])); + } + [Theory] [InlineData(typeof(KernelRuntimeCompatExports), "gettimeofday", "n88vx3C5nW8")] [InlineData(typeof(KernelMemoryCompatExports), "clock_gettime", "lLMT9vJAck0")] diff --git a/tests/SharpEmu.Libs.Tests/Cpu/ImportShutdownTests.cs b/tests/SharpEmu.Libs.Tests/Cpu/ImportShutdownTests.cs new file mode 100644 index 0000000..701e174 --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Cpu/ImportShutdownTests.cs @@ -0,0 +1,115 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Reflection; +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using SharpEmu.Core.Cpu.Native; +using SharpEmu.HLE; +using Xunit; + +namespace SharpEmu.Libs.Tests.Cpu; + +public sealed class ImportShutdownTests +{ + [Theory] + [InlineData("ordinary-import", false, false)] + [InlineData("ordinary-import", true, false)] + [InlineData("BNowx2l588E", true, false)] // Trivial leaf: process counter frequency. + [InlineData("Q3VBxCXhUHs", true, false)] // Hot memory leaf: memcpy. + [InlineData("BNowx2l588E", true, true)] + public void Shutdown_EndsTheSliceWithoutCallingHle(string nid, bool leaf, bool alreadyExiting) + { + using var fixture = new ImportFixture(nid, leaf, shutdown: true, alreadyExiting); + + Assert.Equal(1UL, fixture.Dispatch()); + Assert.Equal(0, fixture.CallCount); + Assert.Equal(1UL, fixture.Context[CpuRegister.Rax]); + Assert.Equal(ImportFixture.HostExit, fixture.ReadImportReturn()); + Assert.True(fixture.Context.TryReadUInt64(ImportFixture.ReturnSlot, out var slot)); + Assert.Equal(ImportFixture.HostExit, slot); + Assert.True(Assert.IsType(ImportFixture.Field("_activeForcedGuestExit").GetValue(null))); + Assert.True(fixture.Context.TryReadUInt64(ImportFixture.Destination, out var destination)); + Assert.Equal(0UL, destination); // No copy may run after service teardown starts. + } + + [Theory] + [InlineData("ordinary-import", true)] + [InlineData("BNowx2l588E", true)] + public void Running_StillCallsTheExport(string nid, bool leaf) + { + using var fixture = new ImportFixture(nid, leaf, shutdown: false, alreadyExiting: false); + + Assert.Equal(0UL, fixture.Dispatch()); + Assert.Equal(1, fixture.CallCount); + Assert.Equal(ImportFixture.GuestReturn, fixture.ReadImportReturn()); + Assert.True(fixture.Context.TryReadUInt64(ImportFixture.ReturnSlot, out var slot)); + Assert.Equal(ImportFixture.GuestReturn, slot); + } + + private sealed class ImportFixture : IDisposable + { + public const ulong ReturnSlot = 0x10000; + public const ulong Destination = 0x10020; + public const ulong HostExit = 0x20000; + public const ulong GuestReturn = 0x30000; + private readonly DirectExecutionBackend _backend; + private readonly Dictionary _previous = new(); + private readonly nint _allocation; + private readonly nint _pack; + public CpuContext Context { get; } + public int CallCount { get; private set; } + + public ImportFixture(string nid, bool leaf, bool shutdown, bool alreadyExiting) + { + _backend = (DirectExecutionBackend)RuntimeHelpers.GetUninitializedObject(typeof(DirectExecutionBackend)); + Context = new CpuContext(new FakeCpuMemory(ReturnSlot, 0x1000), Generation.Gen5); + Assert.True(Context.TryWriteUInt64(ReturnSlot, GuestReturn)); + Assert.True(Context.TryWriteUInt64(Destination + 8, 0xAABBCCDDUL)); + _allocation = Marshal.AllocHGlobal(1024); + Marshal.Copy(new byte[1024], 0, _allocation, 1024); + _pack = _allocation + 512; // Volatile register saves precede the argument pack. + Marshal.WriteInt64(_pack, unchecked((long)Destination)); + Marshal.WriteInt64(_pack + 8, unchecked((long)(Destination + 8))); + Marshal.WriteInt64(_pack + 16, 8); + Marshal.WriteInt64(_pack + 96, unchecked((long)GuestReturn)); + foreach (var name in new[] { "_activeExecutionBackend", "_activeCpuContext", "_activeEntryReturnSentinelRip", + "_activeGuestReturnSlotAddress", "_activeForcedGuestExit", "_activeGuestThreadState", + "_importCounterOwner", "_nextImportDispatchIndex", "_importDispatchBlockEnd" }) + { + var field = Field(name); + _previous.Add(field, field.GetValue(null)); + } + Field("_activeExecutionBackend").SetValue(null, _backend); + Field("_activeCpuContext").SetValue(null, Context); + Field("_activeEntryReturnSentinelRip").SetValue(null, HostExit); + Field("_activeGuestReturnSlotAddress").SetValue(null, ReturnSlot); + Field("_activeForcedGuestExit").SetValue(null, alreadyExiting); + Field("_activeGuestThreadState").SetValue(null, null); + typeof(DirectExecutionBackend).GetField("_forcedGuestExit", BindingFlags.Instance | BindingFlags.NonPublic)! + .SetValue(_backend, shutdown); + var export = new ExportedFunction("test", nid, "test", Generation.Gen5, _ => { CallCount++; return 0; }); + var entryType = typeof(DirectExecutionBackend).GetNestedType("ImportStubEntry", BindingFlags.NonPublic)!; + var entry = Activator.CreateInstance(entryType, [0UL, nid, export, leaf, true, false, false, 1UL]); + var entries = Array.CreateInstance(entryType, 1); + entries.SetValue(entry, 0); + typeof(DirectExecutionBackend).GetField("_importEntries", BindingFlags.Instance | BindingFlags.NonPublic)! + .SetValue(_backend, entries); + } + + public ulong Dispatch() => Assert.IsType(typeof(DirectExecutionBackend).GetMethod("DispatchImport", + BindingFlags.Instance | BindingFlags.NonPublic)!.Invoke(_backend, [0, _pack])); + + public ulong ReadImportReturn() => unchecked((ulong)Marshal.ReadInt64(_pack + 96)); + + public static FieldInfo Field(string name) => typeof(DirectExecutionBackend).GetField(name, + BindingFlags.Static | BindingFlags.NonPublic)!; + + public void Dispose() + { + foreach (var (field, value) in _previous) + field.SetValue(null, value); + Marshal.FreeHGlobal(_allocation); + } + } +} diff --git a/tests/SharpEmu.Libs.Tests/Cpu/PosixSleepStubTests.cs b/tests/SharpEmu.Libs.Tests/Cpu/PosixSleepStubTests.cs new file mode 100644 index 0000000..63b8d20 --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Cpu/PosixSleepStubTests.cs @@ -0,0 +1,27 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Diagnostics; +using SharpEmu.Core.Cpu.Native; +using Xunit; + +namespace SharpEmu.Libs.Tests.Cpu; + +// The POSIX stand-in for kernel32!Sleep takes milliseconds in ecx. It must sleep for a second or +// more as well: macOS usleep rejects such lengths with EINVAL, which made the stub return at once. +public sealed unsafe class PosixSleepStubTests +{ + [NativeX64Theory] + [InlineData(1200u, 1.0, 4.0)] + [InlineData(20u, 0.015, 1.0)] + public void SleepStubSleepsForTheRequestedMilliseconds(uint milliseconds, double minSeconds, double maxSeconds) + { + if (OperatingSystem.IsWindows()) return; + // SysV passes the fourth integer argument in rcx, where the stub reads its milliseconds. + var sleep = (delegate* unmanaged)PosixHostStubs.SleepStubAddress; + var started = Stopwatch.GetTimestamp(); + sleep(0, 0, 0, milliseconds); + var elapsed = Stopwatch.GetElapsedTime(started).TotalSeconds; + Assert.InRange(elapsed, minSeconds, maxSeconds); + } +} diff --git a/tests/SharpEmu.Libs.Tests/GUI/PerGameSettingsTests.cs b/tests/SharpEmu.Libs.Tests/GUI/PerGameSettingsTests.cs index 3237d57..b18bbb2 100644 --- a/tests/SharpEmu.Libs.Tests/GUI/PerGameSettingsTests.cs +++ b/tests/SharpEmu.Libs.Tests/GUI/PerGameSettingsTests.cs @@ -8,6 +8,59 @@ namespace SharpEmu.Libs.Tests.GUI; public sealed class PerGameSettingsTests { + [Fact] + public void GuestResolutionDefaultsToHdAndCanBeOverriddenPerGame() + { + var global = new GuiSettings { Resolution = "3840x2160" }; + Assert.Equal("1920x1080", EffectiveLaunchSettings.Resolve(global, null).GuestResolution); + var game = new PerGameSettings { GuestResolution = "3840x2160" }; + game.RemoveInheritedValues(global); + Assert.False(game.IsEmpty); + var restored = PerGameSettings.NormalizeFromJson(System.Text.Json.JsonSerializer.Serialize(game)); + Assert.Equal("3840x2160", EffectiveLaunchSettings.Resolve(global, restored).GuestResolution); + global.GuestResolution = "3840x2160"; + game.RemoveInheritedValues(global); + Assert.True(game.IsEmpty); + Assert.Equal("3840x2160", EffectiveLaunchSettings.Resolve(global, game).GuestResolution); + } + + [Fact] + public void RenderScaleDefaultsToNativeAndCanBeOverriddenPerGame() + { + var global = new GuiSettings(); + Assert.Equal(1.0f, EffectiveLaunchSettings.Resolve(global, null).RenderScale); + var game = new PerGameSettings { RenderScale = 0.5f }; + game.RemoveInheritedValues(global); + Assert.False(game.IsEmpty); + var restored = PerGameSettings.NormalizeFromJson(System.Text.Json.JsonSerializer.Serialize(game)); + Assert.Equal(0.5f, EffectiveLaunchSettings.Resolve(global, restored).RenderScale); + global.RenderScale = 0.5f; + game.RemoveInheritedValues(global); + Assert.True(game.IsEmpty); + Assert.Equal(0.5f, EffectiveLaunchSettings.Resolve(global, game).RenderScale); + } + + [Theory] + [InlineData(0f, 0.25f)] + [InlineData(100f, 2.0f)] + [InlineData(float.NaN, 1.0f)] + public void OutOfRangeRenderScaleIsClamped(float value, float expected) + { + Assert.Equal(expected, EffectiveLaunchSettings.Resolve( + new GuiSettings(), new PerGameSettings { RenderScale = value }).RenderScale); + } + + [Theory] + [InlineData(null)] + [InlineData("")] + [InlineData("invalid")] + [InlineData("7680x4320")] + public void InvalidGuestResolutionFallsBackToHd(string? value) + { + Assert.Equal("1920x1080", EffectiveLaunchSettings.Resolve( + new GuiSettings(), new PerGameSettings { GuestResolution = value }).GuestResolution); + } + [Fact] public void NewGameInheritsWritableApp0AndGlobalCrashCapture() { diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Buffers/DeviceAddressFaultSpanTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Buffers/DeviceAddressFaultSpanTests.cs new file mode 100644 index 0000000..390d4b8 --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Gpu/Buffers/DeviceAddressFaultSpanTests.cs @@ -0,0 +1,43 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.Libs.Gpu.Buffers; +using Xunit; + +namespace SharpEmu.Libs.Tests.Gpu.Buffers; + +public sealed class DeviceAddressFaultSpanTests +{ + private const ulong Window = GuestBufferCache.DeviceAddressFaultWindow; + private const ulong Page = 0x4000; + + // One fault brings in the whole aligned window around it, not a single page. + [Fact] + public void AFaultInsideALargeMapping_CoversItsAlignedWindow() + { + var mapping = 0x51_0000_0000UL; + var page = mapping + 3 * Window + 0x7C000; + var span = GuestBufferCache.DeviceAddressFaultSpan(page, Page, mapping, 64 * Window); + Assert.Equal(mapping + 3 * Window, span.Address); + Assert.Equal(Window, span.Size); + } + + // The window never reaches past the guest mapping, so it never covers unmapped memory. + [Fact] + public void TheWindowIsClippedToTheMapping() + { + var mapping = 0x51_0000_0000UL + 0x10000; + var span = GuestBufferCache.DeviceAddressFaultSpan(mapping + 0x8000, Page, mapping, 0x30000); + Assert.Equal(mapping, span.Address); + Assert.Equal(0x30000UL, span.Size); + } + + // Without a known mapping the fault keeps its single page. + [Fact] + public void AFaultOutsideAnyMapping_KeepsItsPage() + { + var span = GuestBufferCache.DeviceAddressFaultSpan(0x51_0000_4000UL, Page, 0, 0); + Assert.Equal(0x51_0000_4000UL, span.Address); + Assert.Equal(Page, span.Size); + } +} diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Buffers/GpuBufferTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Buffers/GpuBufferTests.cs index 6a948dd..647b6a7 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Buffers/GpuBufferTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Buffers/GpuBufferTests.cs @@ -17,6 +17,43 @@ public sealed class GpuBufferTests : IClassFixture public GpuBufferTests(HeadlessVulkanFixture fixture) => _vulkan = fixture.Vulkan; + [Fact] + public void SlabPressure_ReleasesIdleChunksButKeepsLiveAndForeignReadBlocks() + { + if (_vulkan is null) return; + using var worker = new CacheWorker(_vulkan); + var info = _vulkan.DeviceInfo; + worker.Run(() => + { + worker.Scheduler.Begin(new SubmissionContext()); + info.Slabs.ReleaseUnused(); + var before = info.Slabs.AllocatedBytes; + const BufferUsageFlags flags = GpuBuffer.AllFlags | BufferUsageFlags.ShaderDeviceAddressBit; + var first = new GpuBuffer(info, worker.Scheduler, GpuBufferUsage.DeviceLocal, 0, flags, 0x1000, allowSlab: true); + var neighbour = new GpuBuffer(info, worker.Scheduler, GpuBufferUsage.DeviceLocal, 0, flags, 0x1000, allowSlab: true); + var allocated = info.Slabs.AllocatedBytes; + Assert.True(allocated > before); + first.Fill(0, first.Size, 0x12345678); + neighbour.Fill(0, neighbour.Size, 0x87654321); + worker.Scheduler.Finish(); + first.Dispose(); + info.Slabs.ReleaseUnused(); + Assert.Equal(allocated, info.Slabs.AllocatedBytes); // The neighbour still owns its block. + using (info.Slabs.BeginForeignRead()) + { + neighbour.Dispose(); + info.Slabs.ReleaseUnused(); + Assert.Equal(allocated, info.Slabs.AllocatedBytes); + } + info.Slabs.ReleaseUnused(); + Assert.Equal(before, info.Slabs.AllocatedBytes); + using var replacement = new GpuBuffer(info, worker.Scheduler, GpuBufferUsage.DeviceLocal, 0, flags, 0x1000, allowSlab: true); + replacement.Fill(0, replacement.Size, 0xABCDEF01); + worker.Scheduler.Finish(); + worker.Scheduler.Shutdown(); + }); + } + [Fact] public void IsInBounds_IsOverflowSafe() { diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Buffers/GuestBufferAsyncReadbackTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Buffers/GuestBufferAsyncReadbackTests.cs index 015ec32..4348661 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Buffers/GuestBufferAsyncReadbackTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Buffers/GuestBufferAsyncReadbackTests.cs @@ -27,7 +27,7 @@ public sealed class GuestBufferAsyncReadbackTests(HeadlessVulkanFixture fixture) [Fact] public void AGuestReadWaitsOffTheQueueThreadAndLandsTheGpuBytes() { - if (_vulkan is null || !GatePrerequisites.Ready(_vulkan)) return; + if (!GatePrerequisites.Ready(_vulkan)) return; using var harness = new CacheHarness(_vulkan); using var fatal = new FatalScope(); Attach(harness, _vulkan); @@ -48,6 +48,190 @@ public sealed class GuestBufferAsyncReadbackTests(HeadlessVulkanFixture fixture) harness.Shutdown(); } + // Without a readback queue the main queue carries the download and the guest thread waits + // for that submission; the worker records the copies and moves on. + [Fact] + public void AGuestReadWithoutAReadbackQueueStillWaitsOffTheQueueThread() + { + if (_vulkan is null || !GatePrerequisites.Ready(_vulkan)) return; + using var harness = new CacheHarness(_vulkan); + using var fatal = new FatalScope(); + Assert.Null(harness.Cache.AsyncReadback); + var address = harness.MapBacked(0x10000, ReadWrite); + GpuWrite(harness, address, 0x55667788); + Assert.True(harness.Cache.HasGpuDirtyBytes(address, 0x100)); + + Assert.True(harness.Cache.DownloadToCpu(address + 8, 4)); + + Assert.False(harness.Cache.HasGpuDirtyBytes(address, 0x100)); + Assert.Equal((1L, 0L), (harness.Cache.PendingReadbacksApplied, harness.Cache.PendingReadbacksRetried)); + Assert.Equal(0x55667788u, BitConverter.ToUInt32(harness.Read(address + 0xF0, 4))); + + // A second write after the first read is picked up by the next read, not served stale. + GpuWrite(harness, address, 0x99AABBCCu); + Assert.True(harness.Cache.DownloadToCpu(address, 4)); + Assert.Equal(0x99AABBCCu, BitConverter.ToUInt32(harness.Read(address + 0x40, 4))); + Assert.Equal((2L, 0L), (harness.Cache.PendingReadbacksApplied, harness.Cache.PendingReadbacksRetried)); + harness.Shutdown(); + } + + // On a unified-memory device a read of a retired GPU write copies straight out of the buffer's + // own mapping: no copy command, no pending readback. + [Fact] + public void AReadOfARetiredWriteComesFromTheUnifiedMapping() + { + if (_vulkan is null || !GatePrerequisites.Ready(_vulkan)) return; + using var harness = new CacheHarness(_vulkan); + using var fatal = new FatalScope(); + if (!harness.Cache.UnifiedBuffers) + { + harness.Shutdown(); + return; + } + + var address = harness.MapBacked(0x10000, ReadWrite); + GpuWrite(harness, address, 0x0BADF00D); + harness.Worker.Run(() => harness.Scheduler.Wait(harness.Scheduler.Flush())); + + Assert.True(harness.Cache.DownloadToCpu(address + 8, 4)); + + Assert.False(harness.Cache.HasGpuDirtyBytes(address, 0x100)); + Assert.Equal(0x0BADF00Du, BitConverter.ToUInt32(harness.Read(address + 0x80, 4))); + Assert.Equal(1L, harness.Cache.MappedReadbacks); + Assert.Equal(0L, harness.Cache.PendingReadbacksApplied); + harness.Shutdown(); + } + + // A read while the writer is submitted but still running waits for that submission only, then + // reads the mapping; a write recorded after the read was issued is picked up by the next read. + [Fact] + public void AReadOfASubmittedWriteWaitsForItAndReadsTheUnifiedMapping() + { + if (_vulkan is null || !GatePrerequisites.Ready(_vulkan)) return; + using var harness = new CacheHarness(_vulkan); + using var fatal = new FatalScope(); + if (!harness.Cache.UnifiedBuffers) + { + harness.Shutdown(); + return; + } + + var address = harness.MapBacked(0x10000, ReadWrite); + GpuWrite(harness, address, 0x12345678); + SubmitBatch(harness); + + Assert.True(harness.Cache.DownloadToCpu(address, 4)); + + Assert.Equal(0x12345678u, BitConverter.ToUInt32(harness.Read(address + 0xFC, 4))); + Assert.Equal(1L, harness.Cache.MappedReadbacks); + Assert.False(harness.Cache.HasGpuDirtyBytes(address, 0x100)); + + GpuWrite(harness, address, 0x9ABCDEF0); + SubmitBatch(harness); + Assert.True(harness.Cache.DownloadToCpu(address, 4)); + Assert.Equal(0x9ABCDEF0u, BitConverter.ToUInt32(harness.Read(address + 0x10, 4))); + harness.Shutdown(); + } + + // A shader store through a device address announces its page only after the fact. When the + // CPU dirtied the page meanwhile, the page stays CPU-owned: the shader's bytes reach guest + // memory by a merge, and the next upload copies only what the CPU changed, so neither the + // earlier nor the later shader bytes in the buffer are overwritten with stale guest bytes. + [Fact] + public void ShaderBytesOnAPageTheCpuAlsoWritesSurviveTheNextUpload() + { + if (_vulkan is null || !GatePrerequisites.Ready(_vulkan)) return; + using var harness = new CacheHarness(_vulkan); + using var fatal = new FatalScope(); + if (!harness.Cache.UnifiedBuffers) + { + harness.Shutdown(); + return; + } + + var address = harness.MapBacked(0x10000, ReadWrite); + harness.Write(address, Enumerable.Repeat((byte)0x11, 0x1000).ToArray()); + var (buffer, offset) = harness.Worker.Run(() => harness.Cache.ObtainBuffer(address, 0x1000, isWritten: false, requiresDeviceAddress: true)); + void ShaderWrite(ulong at, uint value) => harness.Worker.Run(() => + { + buffer.Fill(offset + at, 0x10, value); + harness.Scheduler.Wait(harness.Scheduler.Flush()); + }); + + // The first shader write is reported on a clean page; the CPU write that follows downloads it. + ShaderWrite(0x100, 0x22222222); + harness.Worker.Run(() => harness.Cache.NoteDeviceAddressWrites(address, 0x1000)); + Assert.True(harness.Store.MarkCpuWrite(address + 0x800, 4)); + Assert.Equal(0x22222222u, BitConverter.ToUInt32(harness.Read(address + 0x100, 4))); + harness.Write(address + 0x800, [0xAA, 0xAA, 0xAA, 0xAA]); + + // The second shader write is reported while the page is CPU-dirty, then the page uploads. + ShaderWrite(0x200, 0x33333333); + harness.Worker.Run(() => harness.Cache.NoteDeviceAddressWrites(address, 0x1000)); + Assert.Equal(0x33333333u, BitConverter.ToUInt32(harness.Read(address + 0x200, 4))); + harness.Worker.Run(() => + { + _ = harness.Cache.ObtainBuffer(address, 0x1000, isWritten: false, requiresDeviceAddress: true); + harness.Scheduler.Wait(harness.Scheduler.Flush()); + }); + + var gpu = harness.ReadBack(buffer, offset, 0x1000); + Assert.Equal(0x22222222u, BitConverter.ToUInt32(gpu, 0x100)); + Assert.Equal(0x33333333u, BitConverter.ToUInt32(gpu, 0x200)); + Assert.Equal(0xAAAAAAAAu, BitConverter.ToUInt32(gpu, 0x800)); + Assert.Equal(0x11111111u, BitConverter.ToUInt32(gpu, 0x900)); + harness.Shutdown(); + } + + [Fact] + public void GrowingASharedBufferPreservesShaderBytesNotYetReported() + { + if (_vulkan is null || !GatePrerequisites.Ready(_vulkan)) return; + using var harness = new CacheHarness(_vulkan); + using var fatal = new FatalScope(); + if (!harness.Cache.UnifiedBuffers) + { + harness.Shutdown(); + return; + } + + var address = harness.MapBacked(0x40000, ReadWrite); + harness.Write(address, Enumerable.Repeat((byte)0x11, 0x1000).ToArray()); + var (old, offset) = harness.Worker.Run(() => + harness.Cache.ObtainBuffer(address, 0x1000, isWritten: false, requiresDeviceAddress: true)); + harness.Worker.Run(() => + { + old.Fill(offset + 0x100, 0x10, 0x22222222); + harness.Scheduler.Wait(harness.Scheduler.Flush()); + harness.Cache.NoteDeviceAddressWrites(address, 0x1000); + }); + Assert.True(harness.Store.MarkCpuWrite(address + 0x800, 4)); + harness.Write(address + 0x800, [0xAA, 0xAA, 0xAA, 0xAA]); + harness.Worker.Run(() => + { + _ = harness.Cache.ObtainBuffer(address, 0x1000, isWritten: false, requiresDeviceAddress: true); + harness.Scheduler.Wait(harness.Scheduler.Flush()); + // The device-address fault callback has not reported this new store yet. + old.Fill(offset + 0x200, 0x10, 0x33333333); + harness.Scheduler.Wait(harness.Scheduler.Flush()); + }); + Assert.Equal(0x11111111u, BitConverter.ToUInt32(harness.Read(address + 0x200, 4))); + Assert.True(harness.Store.MarkCpuWrite(address + 0x800, 4)); + harness.Write(address + 0x800, [0xBB, 0xBB, 0xBB, 0xBB]); + var (grown, grownOffset) = harness.Worker.Run(() => + { + var result = harness.Cache.ObtainBuffer(address, 0x20000, isWritten: false, requiresDeviceAddress: true); + harness.Scheduler.Wait(harness.Scheduler.Flush()); + return result; + }); + Assert.NotSame(old, grown); + var gpu = harness.ReadBack(grown, grownOffset, 0x1000); + Assert.Equal(0x22222222u, BitConverter.ToUInt32(gpu, 0x100)); + Assert.Equal(0x33333333u, BitConverter.ToUInt32(gpu, 0x200)); + Assert.Equal(0xBBBBBBBBu, BitConverter.ToUInt32(gpu, 0x800)); + harness.Shutdown(); + } + private static void GpuWrite(CacheHarness harness, ulong address, uint value) => harness.Worker.Run(() => { var (buffer, offset) = harness.Cache.ObtainBuffer(address, 0x100, isWritten: true); @@ -66,7 +250,7 @@ public sealed class GuestBufferAsyncReadbackTests(HeadlessVulkanFixture fixture) [Fact] public void AHotWindowIsReadAheadAfterTheNextGpuWrite() { - if (_vulkan is null || !GatePrerequisites.Ready(_vulkan)) return; + if (!GatePrerequisites.Ready(_vulkan)) return; using var harness = new CacheHarness(_vulkan); using var fatal = new FatalScope(); Attach(harness, _vulkan); @@ -89,7 +273,7 @@ public sealed class GuestBufferAsyncReadbackTests(HeadlessVulkanFixture fixture) [Fact] public void AReadAheadOverwrittenBeforeItsReadIsNeverApplied() { - if (_vulkan is null || !GatePrerequisites.Ready(_vulkan)) return; + if (!GatePrerequisites.Ready(_vulkan)) return; using var harness = new CacheHarness(_vulkan); using var fatal = new FatalScope(); Attach(harness, _vulkan); @@ -110,10 +294,36 @@ public sealed class GuestBufferAsyncReadbackTests(HeadlessVulkanFixture fixture) harness.Shutdown(); } + [Fact] + public void AWindowWhoseReadAheadsKeepGoingStaleStopsGettingThem() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var harness = new CacheHarness(_vulkan); + using var fatal = new FatalScope(); + Attach(harness, _vulkan); + var address = harness.MapBacked(0x10000, ReadWrite); + GpuWrite(harness, address, 0x01010101); + QueueRead(harness, address); + for (var round = 0; round < 4; round++) + { + GpuWrite(harness, address, 0x02020202u + (uint)round); + SubmitBatch(harness); + GpuWrite(harness, address, 0x03030303u + (uint)round); + QueueRead(harness, address); + } + + // Two wasted read-aheads, then the window is skipped; the reads still see the latest bytes. + Assert.Equal(2, harness.Cache.EagerReadbacksStarted); + Assert.Equal(0, harness.Cache.EagerReadbacksUsed); + Assert.Equal(0x03030306u, BitConverter.ToUInt32(harness.Read(address + 0x40, 4))); + Detach(harness); + harness.Shutdown(); + } + [Fact] public void RepeatedGuestReadsReuseTheReadbackSlots() { - if (_vulkan is null || !GatePrerequisites.Ready(_vulkan)) return; + if (!GatePrerequisites.Ready(_vulkan)) return; using var harness = new CacheHarness(_vulkan); using var fatal = new FatalScope(); Attach(harness, _vulkan); diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Buffers/GuestBufferCacheTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Buffers/GuestBufferCacheTests.cs index 5546b62..08ea346 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Buffers/GuestBufferCacheTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Buffers/GuestBufferCacheTests.cs @@ -23,6 +23,72 @@ public sealed class GuestBufferCacheTests : IClassFixture { private const ulong Page = GuestBufferCache.CachingPageSize; + [Fact] + public void SubpageImageUpload_PreservesTheOtherTexturesInItsTrackerPage() + { + if (_vulkan is null) return; + using var fatal = new FatalScope(); + using var harness = new CacheHarness(_vulkan); + var address = harness.MapBacked(0x10000, ReadWrite); + harness.Write(address + 0x100, Bytes(0xffffffffu)); + harness.Write(address + 0x400, Bytes(0xff000000u)); + harness.Worker.Run(() => + { + _ = harness.Cache.FindBuffer(address, 0x10000); + _ = harness.Cache.ObtainBufferForImage(address + 0x400, 256); + }); + var (source, offset) = harness.Worker.Run(() => harness.Cache.ObtainBufferForImage(address + 0x100, 256)); + Assert.Equal(Bytes(0xffffffffu), harness.ReadBufferBytes(source, offset, 4)); + harness.Shutdown(); + } + + [Fact] + public void CommandBackingReadRejectsGpuImageBytesButAllowsCleanPageNeighbours() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var harness = new CacheHarness(_vulkan!); + var address = harness.MapBacked(Page, ReadWrite); + harness.Write(address + 64, new byte[] { 1, 2, 3, 4 }); + var request = Color32(address, 4); + var id = harness.Acquire(ref request); + harness.Worker.Run(() => + { + Assert.True(harness.Images.TryClearImageFromBuffer(address, 16, 0x22222222)); + Assert.True(harness.Image(id).IsGpuModified); + var tick = harness.Scheduler.CurrentTick; + Span bytes = stackalloc byte[4]; + Assert.False(harness.Cache.TryReadCommandBacking(address, bytes)); + Assert.True(harness.Cache.TryReadCommandBacking(address + 64, bytes)); + Assert.Equal(new byte[] { 1, 2, 3, 4 }, bytes.ToArray()); + Assert.Equal(tick, harness.Scheduler.CurrentTick); + }); + } + + [Fact] + public void CommandBytesNextToPendingGpuCounterDoNotWaitOrUnprotectThePage() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var harness = new CacheHarness(_vulkan!); + var address = harness.MapBacked(Page, ReadWrite); + harness.Write(address + 16, new byte[] { 1, 2, 3, 4 }); + harness.Worker.Run(() => + { + var (buffer, offset) = harness.Cache.ObtainBuffer(address, 4, isWritten: true); + buffer.Fill(offset, 4, 123); + var tick = harness.Scheduler.CurrentTick; + var protection = harness.Protection(address); + Span bytes = stackalloc byte[4]; + Assert.True(harness.Cache.TryReadCommandBacking(address + 16, bytes)); + Assert.Equal(new byte[] { 1, 2, 3, 4 }, bytes.ToArray()); + Assert.False(harness.Cache.TryReadCommandBacking(address, bytes)); + Assert.False(harness.Cache.TryReadCommandBacking(address - 2, bytes)); + Assert.Equal(tick, harness.Scheduler.CurrentTick); + Assert.Equal(protection, harness.Protection(address)); + Assert.True(harness.Cache.HasGpuDirtyPages(address, 4)); + }); + } + + [Fact] public void UnalignedImageObtainUploadsTheWholeDirtyPageToItsBufferOwner() { @@ -141,6 +207,77 @@ public sealed class GuestBufferCacheTests : IClassFixture harness.Shutdown(); } + [Theory] + [InlineData(0x80UL)] + [InlineData(0x3F80UL)] + public void DrawAllocationPreparationKeepsShaderVertexAndIndexBindingsInOneLiveBuffer(ulong vertexOffset) + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var harness = new CacheHarness(_vulkan!); + var address = harness.MapBacked(0x40000, ReadWrite); + const uint writtenValue = 0x12345678; + harness.Worker.Run(() => + { + _ = harness.Cache.FindBuffer(address + Page, Page); + var vertexAddress = address + Page + vertexOffset; + harness.Cache.PrepareBufferAllocations([ + new GuestSpan(vertexAddress, Page), new GuestSpan(address, 4 * Page)]); + var (shader, offset) = harness.Cache.ObtainBuffer(address + Page, Page, isWritten: true); + var (vertex, _) = harness.Cache.ObtainBuffer(vertexAddress, Page, isWritten: false); + var (indices, _) = harness.Cache.ObtainBuffer(address, 4 * Page, isWritten: false); + // An attachment readback can finish the preparation tick before the draw's consumer. + harness.Scheduler.Finish(); + Assert.NotEqual(0UL, shader.Handle.Handle); + Assert.Same(shader, vertex); + Assert.Same(shader, indices); + shader.Fill(offset, sizeof(uint), writtenValue); + }); + Assert.True(harness.Cache.TrySynchronizeCpuRead(address + Page, sizeof(uint))); + Assert.Equal(writtenValue, BitConverter.ToUInt32(harness.Read(address + Page, sizeof(uint)))); + harness.Shutdown(); + } + + [Fact] + public void DrawAllocationPreparationLeavesDisjointSmallReadsInTheStreamRing() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var harness = new CacheHarness(_vulkan!); + var address = harness.MapBacked(0x40000, ReadWrite); + harness.Worker.Run(() => + { + harness.Cache.PrepareBufferAllocations([ + new GuestSpan(address, 64), new GuestSpan(address + 2 * Page, 64)]); + Assert.Equal(0, harness.Cache.BufferCount); + var (first, _) = harness.Cache.ObtainBuffer(address, 64, isWritten: false); + var (second, _) = harness.Cache.ObtainBuffer(address + 2 * Page, 64, isWritten: false); + Assert.Same(harness.Cache.GetUtilityBuffer(GpuBufferUsage.Stream), first); + Assert.Same(first, second); + Assert.Equal(0, harness.Cache.BufferCount); + }); + harness.Shutdown(); + } + + [Fact] + public void DrawAllocationPreparationIncludesTouchingVertexRangesBeforeBindingTheShader() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var harness = new CacheHarness(_vulkan!); + var address = harness.MapBacked(0x40000, ReadWrite); + harness.Worker.Run(() => + { + _ = harness.Cache.FindBuffer(address, Page); + harness.Cache.PrepareBufferAllocations([ + new GuestSpan(address, Page), new GuestSpan(address + Page, Page)]); + var (shader, _) = harness.Cache.ObtainBuffer(address, Page, isWritten: true); + // AcquireVertexBuffers merges byte ranges that touch into one request. + var (geometry, _) = harness.Cache.ObtainBuffer(address, 2 * Page, isWritten: false); + harness.Scheduler.Finish(); + Assert.NotEqual(0UL, shader.Handle.Handle); + Assert.Same(shader, geometry); + }); + harness.Shutdown(); + } + [Theory] [InlineData(false)] [InlineData(true)] @@ -279,8 +416,9 @@ public sealed class GuestBufferCacheTests : IClassFixture Assert.Equal(replacementIdentifier, harness.Cache.FindBuffer(address, Page)); Assert.Same(original, harness.Cache.GetBuffer(originalIdentifier)); Assert.NotEqual(0UL, original.Handle.Handle); - Assert.Equal(2 * Page, harness.Cache.TotalUsedMemory); + Assert.Equal(3 * Page, harness.Cache.TotalUsedMemory); harness.Scheduler.Finish(); + Assert.Equal(2 * Page, harness.Cache.TotalUsedMemory); Assert.Equal(0UL, original.Handle.Handle); Assert.NotEqual(0UL, harness.Cache.GetBuffer(replacementIdentifier).Handle.Handle); }); @@ -627,7 +765,12 @@ public sealed class GuestBufferCacheTests : IClassFixture var address = harness.MapBacked(0x10000, ReadWrite); var (buffer, _) = harness.Worker.Run(() => harness.Cache.ObtainBuffer(address, 0x8000, isWritten: false)); - harness.Worker.Run(() => harness.Cache.WriteHostMemory(address + 0x1000, Pattern(0x100, 7))); + harness.Worker.Run(() => + { + harness.Cache.WriteHostMemory(address + 0x1000, Pattern(0x100, 7)); + // Deferred host writes become visible when a command acquires its buffer. + harness.Cache.PrepareBda([new GuestSpan(address, 0x10000)]); + }); Assert.Equal(Pattern(0x100, 7), harness.Read(address + 0x1000, 0x100)); Assert.Equal(Pattern(0x100, 7), harness.ReadBack(buffer, 0x1000, 0x100)); @@ -754,8 +897,8 @@ public sealed class GuestBufferCacheTests : IClassFixture harness.Write(address, expected); harness.Worker.Run(() => { + harness.Cache.NoteMemoryVisibilityPoint(); harness.Cache.PrepareBda([new GuestSpan(address, size)]); - Assert.False(harness.Cache.HasCpuDirtyPages(address, size)); Assert.False(harness.Cache.HasGpuDirtyPages(address, size)); harness.Cache.PrepareBda([new GuestSpan(address, size)]); }); @@ -788,7 +931,11 @@ public sealed class GuestBufferCacheTests : IClassFixture }); Assert.Equal(Bytes(0x12345678u), harness.ReadBack(buffer, offset, 4)); Assert.True(harness.Cache.TrySynchronizeCpuRead(address, size)); - harness.Worker.Run(() => harness.Cache.PrepareBda([new GuestSpan(address, size)])); + harness.Worker.Run(() => + { + harness.Cache.NoteMemoryVisibilityPoint(); + harness.Cache.PrepareBda([new GuestSpan(address, size)]); + }); var expected = Pattern((int)size, 7); await Task.Run(() => @@ -796,12 +943,76 @@ public sealed class GuestBufferCacheTests : IClassFixture Assert.True(harness.Store.MarkCpuWrite(address, size)); harness.Write(address, expected); }); - harness.Worker.Run(() => harness.Cache.PrepareBda([new GuestSpan(address, size)])); + harness.Worker.Run(() => + { + harness.Cache.NoteMemoryVisibilityPoint(); + harness.Cache.PrepareBda([new GuestSpan(address, size)]); + }); Assert.False(harness.Cache.HasCpuDirtyPages(address, size)); Assert.Equal(expected, harness.ReadBack(buffer, offset, size)); harness.Shutdown(); } + // A shader can follow a pointer into ordinary guest memory outside the GPU mappings. + // The fault that reveals it registers the buffer, and preparation must keep touching + // it: otherwise it ages out while in use, faults again, and Ghost of Yotei re-created + // tens of thousands of such buffers, reading zeros in between. + [Fact] + public void DeviceAddressPreparationKeepsFaultedBuffersOutsideTheMappingsResident() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var harness = new CacheHarness(_vulkan); + var mapped = harness.MapBacked(0x20000, ReadWrite); + var pointed = harness.MapBacked(0x20000, ReadWrite); + harness.Worker.Run(() => + { + harness.Cache.RunGarbageCollector(); + harness.Cache.NoteDeviceAddressFault(pointed, 0x8000); + _ = harness.Cache.FindBuffer(pointed, 0x8000); + harness.Cache.SetCollectionThresholds(1, ulong.MaxValue); + for (var collection = 0; collection < 170; collection++) + { + harness.Cache.PrepareBda([new GuestSpan(mapped, 0x20000)]); + harness.Cache.RunGarbageCollector(); + Assert.True(harness.Cache.IsRegionRegistered(pointed, 0x8000)); + harness.Scheduler.Finish(); + } + }); + harness.Shutdown(); + } + + [Fact] + public void DeviceAddressVisibilityPointRefreshesHotPagesWithoutAnotherWriteFault() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var harness = new CacheHarness(_vulkan); + var address = harness.MapBacked(0x20000, ReadWrite); + const ulong size = 0x8000; + var (buffer, offset) = harness.Worker.Run(() => harness.Cache.ObtainBuffer(address, size, false)); + for (var write = 1; write <= 3; write++) + { + Assert.True(harness.Store.MarkCpuWrite(address, size)); + harness.Write(address, Pattern((int)size, (byte)write)); + harness.Worker.Run(() => + { + harness.Cache.NoteMemoryVisibilityPoint(); + harness.Cache.PrepareBda([new GuestSpan(address, size)]); + }); + } + + Assert.True(harness.Cache.HasCpuDirtyPages(address, size)); + var expected = Pattern((int)size, 9); + // A permanently writable hot page changes without another protection fault. + harness.Write(address, expected); + harness.Worker.Run(() => + { + harness.Cache.NoteMemoryVisibilityPoint(); + harness.Cache.PrepareBda([new GuestSpan(address, size)]); + }); + Assert.Equal(expected, harness.ReadBack(buffer, offset, size)); + harness.Shutdown(); + } + [Theory] [InlineData(false)] [InlineData(true)] @@ -1256,4 +1467,76 @@ public sealed class GuestBufferCacheTests : IClassFixture Assert.All(harness.ReadBack(harness.Cache.FaultBuffer, word * 4, 4), value => Assert.Equal(0, value)); harness.Shutdown(); } + + // An async readback waits only for the buffer's last recorded writer, and work submitted + // after it can still reach the buffer through the device-address page table. Collection + // must keep the buffer alive until that work completes: destroying it at once let a + // Ghost of Yotei dispatch write into freed memory (VK_ERROR_DEVICE_LOST, WriteInvalid). + [Fact] + public void GarbageCollector_KeepsAnAsyncDownloadedBufferUntilSubmittedWorkCompletes() + { + if (_vulkan is null) return; + using var harness = new CacheHarness(_vulkan); + var dirty = harness.MapBacked(0x10000, ReadWrite); + var buffer = harness.Worker.Run(() => + { + var (written, offset) = harness.Cache.ObtainBuffer(dirty, 0x100, isWritten: true); + written.Fill(offset, 0x100, 0x0BADF00D); + return written; + }); + harness.Cache.AsyncReadback = new ImmediateReadback(); + harness.Cache.SetCollectionThresholds(1, 1); + harness.Worker.Run(() => + { + harness.Cache.RunGarbageCollector(); + Assert.Equal(0, harness.Cache.BufferCount); + Assert.NotEqual(0UL, buffer.Handle.Handle); + harness.Scheduler.Finish(); + }); + Assert.Equal(0UL, buffer.Handle.Handle); + harness.Cache.AsyncReadback = null; + harness.Shutdown(); + } + + // Hands back zeroed bytes at once, without waiting for the main queue. + private sealed class ImmediateReadback : SharpEmu.Libs.Gpu.Vulkan.IBufferReadback + { + public void Read(ReadOnlySpan pieces, ulong waitTick, + SharpEmu.Libs.Gpu.Vulkan.VulkanAsyncReadback.ReadbackConsumer consume) + { + for (var index = 0; index < pieces.Length; index++) + { + consume(index, new byte[pieces[index].Size]); + } + } + + public void Dispose() + { + } + } + + // The device can run out of memory for a buffer as for an image (Ghost of Tsushima's + // gameplay). The cache frees the images it can and tries the allocation once more. + [Fact] + public void BufferAllocation_RetriesOnceAfterTheDeviceRunsOutOfMemory() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var fatal = new FatalScope(); + const ulong size = 64UL * 1024 * 1024; + using var harness = new CacheHarness(_vulkan, backingBytes: 80UL * 1024 * 1024); + var address = harness.MapBacked(size, ReadWrite); + var attempts = 0; + harness.Vulkan.DeviceInfo.AllocationFailure = bytes => bytes >= size && attempts++ == 0; + try + { + harness.Worker.Run(() => harness.Cache.FindBuffer(address, size)); + Assert.Equal(2, attempts); + } + finally + { + harness.Vulkan.DeviceInfo.AllocationFailure = null; + } + + harness.Shutdown(); + } } diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Buffers/GuestBufferRegistryTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Buffers/GuestBufferRegistryTests.cs index cef0cdc..18d4685 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Buffers/GuestBufferRegistryTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Buffers/GuestBufferRegistryTests.cs @@ -17,6 +17,31 @@ public sealed class GuestBufferRegistryTests public void Dispose() => DisposeCount++; } + [Fact] + public void ResidentBytes_IncludeRetiredAndReplacementUntilEachCompletion() + { + using var registry = new GuestBufferRegistry(16, 4096); + var first = registry.AllocateBuffer(new TestBuffer(), 32, 32); + registry.RegisterBuffer(first, 0); + registry.BeginRetirement(first); + Assert.Equal(0UL, registry.RegisteredBytes); + Assert.Equal(32UL, registry.RetiredBytes); + Assert.Equal(32UL, registry.ResidentBytes); + var second = registry.AllocateBuffer(new TestBuffer(), 32, 64); + registry.RegisterBuffer(second, 1); + Assert.Equal(96UL, registry.ResidentBytes); + Assert.True(registry.CompleteRetirement(first)); + Assert.Equal(64UL, registry.ResidentBytes); + Assert.Equal(0UL, registry.RetiredBytes); + Assert.False(registry.CompleteRetirement(first)); + Assert.Equal(64UL, registry.ResidentBytes); + registry.BeginRetirement(second); + Assert.Equal(64UL, registry.RetiredBytes); + registry.Dispose(); + Assert.Equal(0UL, registry.ResidentBytes); + Assert.Equal(0UL, registry.RetiredBytes); + } + [Fact] public void LifetimeSeparatesLookupRemovalFromResourceRelease() { diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Buffers/OptimalImageMemoryPoolTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Buffers/OptimalImageMemoryPoolTests.cs new file mode 100644 index 0000000..ff09758 --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Gpu/Buffers/OptimalImageMemoryPoolTests.cs @@ -0,0 +1,59 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.Libs.Gpu.Buffers; +using Silk.NET.Vulkan; +using Xunit; + +namespace SharpEmu.Libs.Tests.Gpu.Buffers; + +public sealed class OptimalImageMemoryPoolTests +{ + private sealed class LimitedDevice(ulong budget) : IDeviceMemoryAllocator + { + public readonly List Requests = []; + public readonly Dictionary Allocations = []; + private ulong _next; + public Result AllocateMemory(in MemoryAllocateInfo info, out DeviceMemory memory) + { + Requests.Add(info.AllocationSize); + memory = default; + if (info.AllocationSize > budget - Allocations.Values.Aggregate(0UL, (sum, bytes) => sum + bytes)) + return Result.ErrorOutOfDeviceMemory; + memory = new DeviceMemory(++_next); + Allocations.Add(memory.Handle, info.AllocationSize); + return Result.Success; + } + public void FreeMemory(DeviceMemory memory) => Allocations.Remove(memory.Handle); + } + + [Fact] + public void LargeBlockRejected_SmallImageStillFitsAndIsReleased() + { + var device = new LimitedDevice(32UL << 20); + var pool = new OptimalImageMemoryPool(device); + var requirements = new MemoryRequirements { Size = 22UL << 20, Alignment = 256, MemoryTypeBits = 1 }; + Assert.Equal(Result.Success, pool.Allocate(requirements, 0, out var image)); + Assert.Equal(requirements.Size, pool.AllocatedBytes); + Assert.Equal(requirements.Size, pool.PlacedBytes); + Assert.Equal(new[] { OptimalImageMemoryPool.BlockSize, requirements.Size }, device.Requests); + pool.Free(image); + pool.ReleaseRetained(); + Assert.Empty(device.Allocations); + Assert.Equal(0UL, pool.AllocatedBytes); + Assert.Equal(0UL, pool.PlacedBytes); + } + + [Fact] + public void ActualImageCannotFit_FailureLeavesPoolAccountingUnchanged() + { + var device = new LimitedDevice(1UL << 20); + var pool = new OptimalImageMemoryPool(device); + var requirements = new MemoryRequirements { Size = 22UL << 20, Alignment = 256, MemoryTypeBits = 1 }; + Assert.Equal(Result.ErrorOutOfDeviceMemory, pool.Allocate(requirements, 0, out _)); + Assert.Empty(device.Allocations); + Assert.Equal(0UL, pool.AllocatedBytes); + Assert.Equal(0UL, pool.PlacedBytes); + Assert.Equal(0, pool.Placements); + } +} diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Buffers/SharedPageShadowsTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Buffers/SharedPageShadowsTests.cs new file mode 100644 index 0000000..dff53f4 --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Gpu/Buffers/SharedPageShadowsTests.cs @@ -0,0 +1,207 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.Libs.Gpu.Buffers; +using Silk.NET.Vulkan; +using Xunit; + +namespace SharpEmu.Libs.Tests.Gpu.Buffers; + +public sealed class SharedPageShadowsTests +{ + private const ulong PageBytes = 64; + private const ulong Page = 0x1000; + + [Fact] + public void UploadCopiesOnlyTheBytesTheCpuChanged() + { + var owner = new object(); + var shadows = new SharedPageShadows(PageBytes, 4); + var gpu = Fill(0x11); + Assert.True(shadows.Adopt(Page, owner, gpu)); + + // The CPU changed bytes 4..7 and 40; the shader's bytes elsewhere must survive the upload. + var staged = Fill(0x11); + staged.AsSpan(4, 4).Fill(0xAA); + staged[40] = 0xBB; + var copies = new List { new(0, 0, PageBytes) }; + shadows.FilterUpload(copies, owner, Page, staged, tick: 3); + + Assert.Equal([new BufferCopy(4, 4, 4), new BufferCopy(40, 40, 1)], copies); + Assert.Equal(PageBytes - 5, (ulong)shadows.SkippedUploadBytes); + } + + [Fact] + public void UploadKeepsCopiesOutsideSharedPagesWhole() + { + var owner = new object(); + var shadows = new SharedPageShadows(PageBytes, 4); + Assert.True(shadows.Adopt(Page + PageBytes, owner, Fill(0))); + + // One copy covers the page before, the shared page and the page after; it stages at 0x100. + var staged = new byte[0x100 + 3 * PageBytes]; + staged[0x100 + PageBytes + 9] = 7; + var copies = new List { new(0x100, 0, 3 * PageBytes), new(0, 0x500, 8) }; + shadows.FilterUpload(copies, owner, Page, staged, tick: 1); + + Assert.Equal( + [ + new BufferCopy(0x100, 0, PageBytes), + new BufferCopy(0x100 + PageBytes + 9, PageBytes + 9, 1), + new BufferCopy(0x100 + 2 * PageBytes, 2 * PageBytes, PageBytes), + new BufferCopy(0, 0x500, 8), + ], + copies); + } + + [Fact] + public void UploadIntoANewOwnerCopiesTheWholePage() + { + var shadows = new SharedPageShadows(PageBytes, 4); + Assert.True(shadows.Adopt(Page, new object(), Fill(0x11))); + var copies = new List { new(0, 0, PageBytes) }; + shadows.FilterUpload(copies, new object(), Page, Fill(0x11), tick: 1); + Assert.Equal([new BufferCopy(0, 0, PageBytes)], copies); + } + + [Fact] + public void PullMergesShaderBytesWhereTheCpuLeftThemAlone() + { + var owner = new object(); + var shadows = new SharedPageShadows(PageBytes, 4); + Assert.True(shadows.Adopt(Page, owner, Fill(0x11))); + + // The shader wrote bytes 8..15; the CPU meanwhile wrote byte 10 (it keeps it) and byte 50. + var gpu = Fill(0x11); + gpu.AsSpan(8, 8).Fill(0x22); + var guest = Fill(0x11); + guest[10] = 0x33; + guest[50] = 0x44; + var writes = new List<(int Offset, int Length)>(); + var merged = shadows.PullGpuWrites(Page, owner, gpu, guest, completedTick: 0, (offset, length) => writes.Add((offset, length))); + + Assert.Equal(7, merged); + Assert.Equal([(8, 2), (11, 5)], writes); + var expected = Fill(0x11); + expected.AsSpan(8, 8).Fill(0x22); + expected[10] = 0x33; + expected[50] = 0x44; + Assert.Equal(expected, guest); + + // The next upload carries both CPU bytes, including the one the shader also wrote. + var copies = new List { new(0, 0, PageBytes) }; + shadows.FilterUpload(copies, owner, Page, guest, tick: 1); + Assert.Equal([new BufferCopy(10, 10, 1), new BufferCopy(50, 50, 1)], copies); + } + + [Fact] + public void PullWaitsForStagedUploadsToExecute() + { + var owner = new object(); + var shadows = new SharedPageShadows(PageBytes, 4); + Assert.True(shadows.Adopt(Page, owner, Fill(0))); + var staged = Fill(0); + staged[3] = 9; + var copies = new List { new(0, 0, PageBytes) }; + shadows.FilterUpload(copies, owner, Page, staged, tick: 5); + + // The GPU copy does not hold the staged byte yet; it must not read as a shader change. + var guest = staged.ToArray(); + Assert.Equal(0, shadows.PullGpuWrites(Page, owner, Fill(0), guest, completedTick: 4, (_, _) => { })); + Assert.Equal(9, guest[3]); + var gpu = staged.ToArray(); + gpu[20] = 1; + Assert.Equal(1, shadows.PullGpuWrites(Page, owner, gpu, guest, completedTick: 5, (_, _) => { })); + Assert.Equal(9, guest[3]); + Assert.Equal(1, guest[20]); + } + + [Fact] + public void ADownloadMakesTheShadowAgreeWithTheGpuCopy() + { + var owner = new object(); + var shadows = new SharedPageShadows(PageBytes, 4); + Assert.True(shadows.Adopt(Page, owner, Fill(0))); + + // The shader writes bytes 16..23 later; the CPU write that follows downloads them first. + var downloaded = new byte[8]; + Array.Fill(downloaded, (byte)0x55); + shadows.NoteDownloaded(Page + 16, downloaded); + var staged = Fill(0); + staged.AsSpan(16, 8).Fill(0x55); + staged[30] = 0x66; + var copies = new List { new(0, 0, PageBytes) }; + shadows.FilterUpload(copies, owner, Page, staged, tick: 2); + Assert.Equal([new BufferCopy(30, 30, 1)], copies); + } + + [Theory] + [InlineData(0UL)] + [InlineData(3UL)] + public void AnEliminatedUploadDoesNotDelayCompletedShaderWrites(ulong lastUploadTick) + { + var owner = new object(); + var shadows = new SharedPageShadows(PageBytes, 4); + var guest = Fill(0); + Assert.True(shadows.Adopt(Page, owner, guest)); + if (lastUploadTick != 0) + { + guest[3] = 9; + var first = new List { new(0, 0, PageBytes) }; + shadows.FilterUpload(first, owner, Page, guest, lastUploadTick); + Assert.Equal([new BufferCopy(3, 3, 1)], first); + } + + // Hot CPU pages can be offered for upload on every draw, even when no bytes changed. + // Filtering removes this upload entirely: its future tick must not gate a GPU merge. + var copies = new List { new(0, 0, PageBytes) }; + shadows.FilterUpload(copies, owner, Page, guest, tick: 5); + Assert.Empty(copies); + var gpu = guest.ToArray(); + gpu[20] = 0x55; + var writes = new List<(int Offset, int Length)>(); + Assert.Equal(1, shadows.PullGpuWrites(Page, owner, gpu, guest, lastUploadTick, + (offset, length) => writes.Add((offset, length)))); + Assert.Equal([(20, 1)], writes); + Assert.Equal(0x55, guest[20]); + } + + [Fact] + public void AdoptStopsAtCapacity() + { + var shadows = new SharedPageShadows(PageBytes, 1); + Assert.True(shadows.Adopt(Page, new object(), Fill(0))); + Assert.False(shadows.Adopt(Page + PageBytes, new object(), Fill(0))); + Assert.True(shadows.Contains(Page)); + Assert.False(shadows.Contains(Page + PageBytes)); + } + + [Fact] + public void ABufferCopyTransfersTheBaselineAndWaitsForTheCopy() + { + var old = new object(); + var replacement = new object(); + var shadows = new SharedPageShadows(PageBytes, 4); + Assert.True(shadows.Adopt(Page, old, Fill(0x11))); + shadows.NoteOwnerCopy(old, replacement, tick: 5); + var guest = Fill(0x11); + guest[40] = 0xAA; + var copies = new List { new(0, 0, PageBytes) }; + shadows.FilterUpload(copies, replacement, Page, guest, tick: 5); + Assert.Equal([new BufferCopy(40, 40, 1)], copies); + var gpu = Fill(0x11); + gpu[20] = 0x55; + gpu[40] = 0xAA; + Assert.Equal(0, shadows.PullGpuWrites(Page, replacement, gpu, guest, 4, (_, _) => { })); + Assert.Equal(1, shadows.PullGpuWrites(Page, replacement, gpu, guest, 5, (_, _) => { })); + Assert.Equal(0x55, guest[20]); + Assert.Equal(0xAA, guest[40]); + } + + private static byte[] Fill(byte value) + { + var bytes = new byte[PageBytes]; + Array.Fill(bytes, value); + return bytes; + } +} diff --git a/tests/SharpEmu.Libs.Tests/Gpu/GpuCommands/CommandStreamQueueTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/GpuCommands/CommandStreamQueueTests.cs index b942cbf..ae65d04 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/GpuCommands/CommandStreamQueueTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/GpuCommands/CommandStreamQueueTests.cs @@ -332,6 +332,22 @@ public sealed class CommandStreamQueueTests Assert.Equal(9u, queue.GetInterpreter(0).InstanceCount); } + [Fact] + public void DrainForShutdown_DropsPendingWorkOnceTheGuestMemoryIsGone() + { + var (host, queue) = NewQueue(); + Enqueue(host, queue, Graphics, 1, CreateInstanceCountPacket(9)); + EnqueueCompute(host, queue, 0x20, Compute, 2, CreateInstanceCountPacket(5)); + queue.StopAccepting(); + + var outcome = queue.DrainForShutdown(cancelBlockedOnNoProgress: true, abandon: () => true); + + Assert.Equal(IdleOutcome.Cancelled, outcome); + Assert.NotEqual(9u, queue.GetInterpreter(0).InstanceCount); + Assert.False(queue.HasPending); + Assert.Equal(IdleOutcome.Cancelled, queue.WaitForIdle()); + } + [Fact] public void DrainForShutdown_CancelsOnlyHeadsThatMakeNoProgressAndTheOutcomeSticks() { diff --git a/tests/SharpEmu.Libs.Tests/Gpu/GpuCommands/CommandStreamTestSupport.cs b/tests/SharpEmu.Libs.Tests/Gpu/GpuCommands/CommandStreamTestSupport.cs index c61697d..9761146 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/GpuCommands/CommandStreamTestSupport.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/GpuCommands/CommandStreamTestSupport.cs @@ -31,6 +31,11 @@ internal sealed class RecordingCommandStreamHost : ICommandStreamHost public List EndOfPipeWrites { get; } = new(); + // Satisfied waits and command-processor memory writes reported to the host. + public int VisibilityPoints { get; private set; } + + public void NoteMemoryVisibilityPoint() => VisibilityPoints++; + public List GuestReads { get; } = new(); public Action? BeforeGuestRead { get; set; } @@ -180,6 +185,8 @@ internal sealed class RecordingCommandStreamHost : ICommandStreamHost public bool ResolvesIndirectDrawOnGpu { get; set; } + public bool ResolvesNonIndexedIndirectDrawOnGpu { get; set; } + public void OnQueueReset(int queueId) => Calls.Add($"queue_reset {queueId}"); public Exception Fatal(string message) => new CommandStreamFatalException(message); diff --git a/tests/SharpEmu.Libs.Tests/Gpu/GpuCommands/CommandStreamWorkerTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/GpuCommands/CommandStreamWorkerTests.cs index 49b8a3f..ae04a3c 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/GpuCommands/CommandStreamWorkerTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/GpuCommands/CommandStreamWorkerTests.cs @@ -48,17 +48,25 @@ public sealed class CommandStreamWorkerTests [Fact] public void Worker_RetriesABlockedHeadAfterTheTimedWait() { - var (host, queue, worker) = NewWorker(cancelBlockedAtStop: false); + var (host, queue, worker) = NewWorker(cancelBlockedAtStop: true); host.WriteDword(Label, 0); - Enqueue(host, queue, 1, WaitEqual(Label, 1), CreateInstanceCountPacket(6)); - Thread.Sleep(50); - host.WriteDword(Label, 1); + try + { + Enqueue(host, queue, 1, WaitEqual(Label, 1), CreateInstanceCountPacket(6)); + // A loaded runner can start the worker after a fixed sleep has elapsed. + // Publish the signal only once the worker has actually blocked on it. + Assert.True(SpinWait.SpinUntil(() => queue.BlockedQueueCount == 1, TimeSpan.FromSeconds(5))); + host.WriteDword(Label, 1); - Assert.Equal(IdleOutcome.Completed, queue.WaitForIdle()); - Assert.Equal(6u, queue.GetInterpreter(0).InstanceCount); - Assert.True(queue.BlockedRetries >= 1); - worker.Stop(); + Assert.Equal(IdleOutcome.Completed, queue.WaitForIdle()); + Assert.Equal(6u, queue.GetInterpreter(0).InstanceCount); + Assert.True(queue.BlockedRetries >= 1); + } + finally + { + worker.Stop(); + } } [Fact] diff --git a/tests/SharpEmu.Libs.Tests/Gpu/GpuCommands/GpuCommandInterpreterDrawTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/GpuCommands/GpuCommandInterpreterDrawTests.cs index 707e5b0..298269d 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/GpuCommands/GpuCommandInterpreterDrawTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/GpuCommands/GpuCommandInterpreterDrawTests.cs @@ -126,6 +126,24 @@ public sealed class GpuCommandInterpreterDrawTests Assert.Contains("arguments base is zero", new StreamRunner().RunExpectingFatal(StreamRunner.Packet(PacketOpcode.DrawIndirect, 0, 0, 0, 2)).Message); } + [Fact] + public void NonIndexedIndirectDraws_LeaveArgumentsOnTheGpuWhenTheHostCan() + { + var runner = new StreamRunner(); + runner.Host.ResolvesNonIndexedIndirectDrawOnGpu = true; + runner.Host.WriteWords(Arguments, new uint[] { 9, 3, 1, 2 }); + + runner.Run(SetBase(Arguments, dispatch: false), StreamRunner.Packet(PacketOpcode.DrawIndirect, 0, 0, 0, 2)); + + var auto = Assert.Single(runner.Host.AutoDraws); + Assert.Equal((Arguments, DrawOffsetSource.IndirectArguments), (auto.IndirectArgumentsAddress, auto.OffsetSource)); + Assert.DoesNotContain(runner.Host.GuestReads, address => address >= Arguments && address < Arguments + 16); + + // A later draw that inherits the instance count reads it then. + runner.Run(StreamRunner.Packet(PacketOpcode.DrawIndexAuto, 5, 0)); + Assert.Equal(3u, runner.Host.AutoDraws[1].InstanceCount); + } + [Fact] public void IndexedIndirectDraws_LeaveArgumentsOnTheGpuWhenTheHostCan() { diff --git a/tests/SharpEmu.Libs.Tests/Gpu/GpuCommands/GpuCommandInterpreterLabelTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/GpuCommands/GpuCommandInterpreterLabelTests.cs index 1425f35..a79f769 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/GpuCommands/GpuCommandInterpreterLabelTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/GpuCommands/GpuCommandInterpreterLabelTests.cs @@ -325,12 +325,16 @@ public sealed class GpuCommandInterpreterLabelTests Assert.Contains("event type is unknown", runner.RunExpectingFatal(StreamRunner.Packet(PacketOpcode.EventWrite, 0x3F)).Message); runner.Run(StreamRunner.Packet(PacketOpcode.EventWrite, 0x39 | (1u << 8), StreamRunner.Low(Label), StreamRunner.High(Label))); + // CPU-only stores leave GPU query-reduction inputs stale. Require the + // host transfer path for both words of every counter, including ready. + Assert.Equal(32, runner.Host.Calls.Count); + Assert.All(runner.Host.Calls, call => Assert.StartsWith("fill ", call)); Assert.Equal(1UL << 63, runner.Host.ReadQword(Label)); Assert.Equal(1UL << 63, runner.Host.ReadQword(Label + 15 * 16)); Assert.Equal(0UL, runner.Host.ReadQword(Label + 8)); runner.Run(StreamRunner.Packet(PacketOpcode.EventWrite, 0x39 | (1u << 8), StreamRunner.Low(Label), StreamRunner.High(Label))); - Assert.Equal((1UL << 63) | 1, runner.Host.ReadQword(Label)); + Assert.Equal((1UL << 63) | 0x2FFFFFFUL, runner.Host.ReadQword(Label)); } [Fact] diff --git a/tests/SharpEmu.Libs.Tests/Gpu/GpuCommands/GpuCommandInterpreterLoopTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/GpuCommands/GpuCommandInterpreterLoopTests.cs index 4c93b35..c74ce4f 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/GpuCommands/GpuCommandInterpreterLoopTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/GpuCommands/GpuCommandInterpreterLoopTests.cs @@ -261,6 +261,30 @@ public sealed class GpuCommandInterpreterLoopTests Assert.Contains(Label, runner.Host.GuestReads); } + [Fact] + public void Wait_IsAMemoryVisibilityPointOnlyOnceSatisfied() + { + var runner = new StreamRunner(); + + Assert.Equal(SubmissionProgress.Blocked, runner.Run(WaitEqual(Label, 1))); + Assert.Equal(0, runner.Host.VisibilityPoints); + + runner.Host.WriteDword(Label, 1); + Assert.Equal(SubmissionProgress.Complete, runner.Run()); + Assert.Equal(1, runner.Host.VisibilityPoints); + } + + [Fact] + public void WriteData_IsAMemoryVisibilityPoint() + { + var runner = new StreamRunner(); + + runner.Run(WriteData(Label, 7)); + + Assert.Equal(7u, runner.Host.ReadDword(Label)); + Assert.Equal(1, runner.Host.VisibilityPoints); + } + [Fact] public void PendingCommands_RunBeforeEveryPacket() { diff --git a/tests/SharpEmu.Libs.Tests/Gpu/GpuCommands/Registers/RegisterBanksTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/GpuCommands/Registers/RegisterBanksTests.cs index db22282..ade0350 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/GpuCommands/Registers/RegisterBanksTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/GpuCommands/Registers/RegisterBanksTests.cs @@ -10,6 +10,17 @@ namespace SharpEmu.Libs.Tests.Gpu.GpuCommands.Registers; // The context lifecycle over the typed banks: push copies, pop restores, clear returns to the defaults. public sealed class RegisterBanksTests { + [Theory] + [InlineData(0x00000110u, false)] + [InlineData(0x00001110u, true)] + public void DepthBeforeShaderIsIndependentOfEarlyThenLateOrderAndMaskExport(uint raw, bool forcedEarly) + { + var control = DepthShaderControlRegisters.Decode(raw); + Assert.Equal((byte)1, control.DepthExportOrder); + Assert.True(control.MaskExportEnable); + Assert.Equal(forcedEarly, control.DepthBeforeShader); + Assert.Equal(0u, control.RemainingBits); + } private static RegisterBanks NewBanks() => new(static message => new InvalidOperationException(message)); [Fact] diff --git a/tests/SharpEmu.Libs.Tests/Gpu/GpuCommands/Registers/RegisterWriteTableTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/GpuCommands/Registers/RegisterWriteTableTests.cs index c93f259..cce4001 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/GpuCommands/Registers/RegisterWriteTableTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/GpuCommands/Registers/RegisterWriteTableTests.cs @@ -32,6 +32,151 @@ public sealed class RegisterWriteTableTests private static uint Float(float value) => BitConverter.SingleToUInt32Bits(value); + [Fact] + public void HullUserDataAddress_PreservesSeparateLowHighRegistersAndCopiedState() + { + var banks = NewBanks(); + WriteShader(banks, SpiShaderUserDataHs0, 0x1111, 0x20); + Assert.Equal(2u, WriteShader(banks, SpiShaderUserDataAddrLoHs, 0x1234_56BC, 0xFFFF_ABCD)); + Assert.Equal(0xABCD_1234_56BCul, banks.Shader.Vertex.HullUserDataAddress); + var copy = banks.Shader.Vertex.Copy(); + RegisterWriteTable.WriteShaderEntry(banks, SpiShaderUserDataAddrHiHs, 0x23, PacketAddress); + Assert.Equal(0x23_1234_56BCul, banks.Shader.Vertex.HullUserDataAddress); + RegisterWriteTable.WriteShaderEntry(banks, SpiShaderUserDataAddrLoHs, 0x8765_4304, PacketAddress); + Assert.Equal(0x23_8765_4304ul, banks.Shader.Vertex.HullUserDataAddress); + Assert.Equal(0xABCD_1234_56BCul, copy.HullUserDataAddress); + Assert.Equal(0x1111u, banks.Shader.Vertex.HullUserScalars.Values[0]); + } + + [Fact] + public void UndocumentedShaderRegister192_IsAcceptedFromTablesAndPackets() + { + var banks = NewBanks(); + var vertex = banks.Shader.Vertex.Copy(); + RegisterWriteTable.WriteShaderEntry(banks, SpiShaderUndocumented192, 1, PacketAddress); + Assert.Equal(1u, WriteShader(banks, SpiShaderUndocumented192, 1)); + Assert.Equal(vertex.LocalAddress, banks.Shader.Vertex.LocalAddress); + Assert.Equal(vertex.HullAddress, banks.Shader.Vertex.HullAddress); + } + + [Fact] + public void UndocumentedUserConfigRegister191_IsAcceptedFromTablesAndPackets() + { + var banks = NewBanks(); + var primitiveType = banks.UserConfig.PrimitiveType; + RegisterWriteTable.WriteUserConfigEntry(banks, UserConfigUndocumented191, 0, PacketAddress); + Assert.Equal(1u, WriteUserConfig(banks, UserConfigUndocumented191, 0)); + Assert.Equal(primitiveType, banks.UserConfig.PrimitiveType); + } + + [Fact] + public void CapturedMixedSampleRegistersKeepDepthStorageSeparateFromEqaaAnchors() + { + var banks = NewBanks(); + WriteContext(banks, DbZInfo, 0xA0000187); + WriteContext(banks, CbColor0Attrib, 0); + WriteContext(banks, PaScAaConfig, 0x00108001); + WriteContext(banks, DbEqaa, 0); + + // DB_Z_INFO.NUM_SAMPLES is bits 3:2; CB_COLOR0_ATTRIB has + // coverage bits 14:12 and stored fragment bits 16:15. + Assert.Equal(2u, 1u << (int)banks.Context.DepthTarget.SamplesLog2); + Assert.Equal(1u, 1u << (int)banks.Context.ColorTargets[0].SamplesLog2); + Assert.Equal(1u, 1u << (int)banks.Context.ColorTargets[0].FragmentsLog2); + Assert.Equal(2u, 1u << banks.Context.AntialiasingConfig.SampleCountLog2); + Assert.Equal(2u, 1u << banks.Context.AntialiasingConfig.ExposedSamplesLog2); + Assert.Equal(0, banks.Context.EnhancedQualityAntialiasing.MaxAnchorSamples); + + // Anchor association state does not change the depth allocation's sample count. + WriteContext(banks, DbEqaa, 1); + Assert.Equal(1, banks.Context.EnhancedQualityAntialiasing.MaxAnchorSamples); + Assert.Equal(2u, 1u << (int)banks.Context.DepthTarget.SamplesLog2); + } + + [Fact] + public void ScanModeControl1WritesPreserveSampleIterationAndOtherBits() + { + var banks = NewBanks(); + Assert.Equal(1u, WriteContext(banks, PaScModeCntl1, 0x08010001)); + Assert.Equal(0x08010001u, banks.Context.ScanModeControl1); + RegisterWriteTable.ContextIndirect[PaScModeCntl1]!(banks, PaScModeCntl1, 0x00000001); + Assert.Equal(0x00000001u, banks.Context.ScanModeControl1); + } + + [Fact] + public void EqaaWritesPreserveIndependentRatesAndDepthControls() + { + var banks = NewBanks(); + const uint value = 0x0D3F4321; + Assert.Equal(1u, WriteContext(banks, DbEqaa, value)); + var eqaa = banks.Context.EnhancedQualityAntialiasing; + Assert.Equal(1, eqaa.MaxAnchorSamples); + Assert.Equal(2, eqaa.PixelShaderIterationSamples); + Assert.Equal(3, eqaa.MaskExportSamples); + Assert.Equal(4, eqaa.AlphaToMaskSamples); + Assert.True(eqaa.HighQualityIntersections); + Assert.True(eqaa.IncoherentReads); + Assert.True(eqaa.InterpolateComponentZ); + Assert.True(eqaa.InterpolateSourceZ); + Assert.True(eqaa.StaticAnchorAssociations); + Assert.True(eqaa.AlphaToMaskEqaaDisable); + Assert.Equal(5, eqaa.OverrasterizationAmount); + Assert.True(eqaa.EnablePostZOverrasterization); + + Assert.Equal(1u, WriteContext(banks, DbEqaa, 0)); + eqaa = banks.Context.EnhancedQualityAntialiasing; + Assert.False(eqaa.InterpolateSourceZ); + Assert.False(eqaa.AlphaToMaskEqaaDisable); + Assert.Equal(0, eqaa.OverrasterizationAmount); + Assert.False(eqaa.EnablePostZOverrasterization); + } + + [Fact] + public void SampleMaskWritesPreserveValuesAndFollowingRegisters() + { + var banks = NewBanks(); + Assert.Equal(1u, WriteContext(banks, PsShaderSampleExclusionMask, 0xFFFE)); + Assert.Equal(2u, WriteContext(banks, PaScAaMaskX0Y0X1Y0, 0x00010002, 0x00040008)); + Assert.Equal(0xFFFEu, banks.Context.ShaderSampleExclusionMask); + Assert.Equal(0x00010002u, banks.Context.SampleCoverageMaskX0Y0X1Y0); + Assert.Equal(0x00040008u, banks.Context.SampleCoverageMaskX0Y1X1Y1); + } + + [Theory] + [InlineData(0x40u, true)] + [InlineData(0u, false)] + [InlineData(0x68000040u, true)] + public void DepthOverridePreservesForcedShaderOrderAndResetsItOnLaterWrites(uint raw, bool forced) + { + var banks = NewBanks(); + Assert.Equal(1u, WriteContext(banks, DbRenderOverride, raw)); + Assert.Equal(forced, banks.Context.DepthRenderOverride.ForceShaderDepthOrder); + Assert.Equal(forced, banks.Context.Copy().DepthRenderOverride.ForceShaderDepthOrder); + Assert.Equal((raw & 0x20000000u) != 0, banks.Context.DepthRenderOverride.ForceZValid); + Assert.Equal((raw & 0x08000000u) != 0, banks.Context.DepthRenderOverride.ForceZDirty); + Assert.Equal((raw & 0x40000000u) != 0, banks.Context.DepthRenderOverride.ForceStencilValid); + Assert.Equal(1u, WriteContext(banks, DbRenderOverride, 0)); + Assert.False(banks.Context.DepthRenderOverride.ForceShaderDepthOrder); + } + + [Fact] + public void SamplePositionsDecodeSignedOffsetsAndQuadPixels() + { + var locations = new SampleLocationRegisters(); + locations.Locations[0] = 0x00000C04; + locations.Locations[4] = 0x0000040C; + locations.Locations[8] = 0x00000088; + locations.Locations[15] = 0x77000000; + Assert.Equal((0.75f, 0.5f), locations.Position(0, 0)); + Assert.Equal((0.25f, 0.5f), locations.Position(0, 1)); + Assert.Equal((0.25f, 0.5f), locations.Position(1, 0)); + Assert.Equal((0.75f, 0.5f), locations.Position(1, 1)); + Assert.Equal((0f, 0f), locations.Position(2, 0)); + Assert.Equal((0.9375f, 0.9375f), locations.Position(3, 15)); + Assert.Throws(() => locations.Position(4, 0)); + Assert.Throws(() => locations.Position(0, 16)); + } + [Fact] public void DirectWriters_DecodeDepthControlAndModeControl() { diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Images/CachedImageTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Images/CachedImageTests.cs index 1b84c97..db89999 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Images/CachedImageTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Images/CachedImageTests.cs @@ -66,14 +66,16 @@ public sealed unsafe class CachedImageTests : IClassFixture 0); var unbacked = Color2D(8, 8); unbacked.Data = GuestSpan.Empty; unbacked.Pitch = 0; var fresh = harness.CreateImage(unbacked); Assert.False(fresh.IsCpuDirty); - Assert.Equal(0UL, fresh.AccountedSize); + Assert.Equal(fresh.Backing.AllocationSize, fresh.AccountedSize); + Assert.True(fresh.AccountedSize > 0); var association = ImageDescription.Create(); association.Data = new GuestSpan(ArrayBackedSpace.Base, 0x100); @@ -203,6 +205,27 @@ public sealed unsafe class CachedImageTests : IClassFixture } } + [Fact] + public void ScratchPressure_TrimsOnlyCompletedBuffers() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var harness = new ImageTestHarness(_vulkan); + var baseline = harness.Device.LiveAllocations; + harness.Run(() => + { + _ = harness.Tiler.GetScratchBuffer(64); + Assert.Equal(baseline + 1, harness.Device.LiveAllocations); + harness.Tiler.ReleaseUnusedScratch(); + Assert.Equal(baseline + 1, harness.Device.LiveAllocations); + Assert.True(harness.Tiler.OutstandingScratchBytes > 0); + harness.Scheduler.Finish(); + Assert.Equal(0UL, harness.Tiler.OutstandingScratchBytes); + Assert.True(harness.Tiler.PooledScratchBytes > 0); + harness.Tiler.ReleaseUnusedScratch(); + Assert.Equal(baseline, harness.Device.LiveAllocations); + Assert.Equal(0UL, harness.Tiler.PooledScratchBytes); + _ = harness.Tiler.GetScratchBuffer(64); + harness.Scheduler.Finish(); + }); + harness.AssertNoValidationMessages(); + } + [Fact] // Scratches go back to a pool when their tick completes instead of being freed, so // the next tick's scratches of the same sizes reuse them without new allocations. diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Images/GuestImageCacheTests.Depth.cs b/tests/SharpEmu.Libs.Tests/Gpu/Images/GuestImageCacheTests.Depth.cs index 1362d40..eac4904 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Images/GuestImageCacheTests.Depth.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Images/GuestImageCacheTests.Depth.cs @@ -608,4 +608,154 @@ public sealed unsafe partial class GuestImageCacheTests Assert.All(Stencil(depthId), value => Assert.Equal(0, value)); harness.Shutdown(); } + + [Fact] + public void SmallerImageAtTheStencilAddress_IsNotTakenAsThePlane() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + const ImageUsageFlags usage = ImageUsageFlags.TransferSrcBit | ImageUsageFlags.TransferDstBit | ImageUsageFlags.DepthStencilAttachmentBit; + if (!_vulkan.DeviceInfo.TryGetImageFormatProperties(Format.D32SfloatS8Uint, ImageType.Type2D, ImageTiling.Optimal, usage, 0, out _)) return; + using var harness = new CacheHarness(_vulkan); + const uint width = 512; + const ulong depthSize = width * width * 4; + const ulong plane = width * width; + var address = harness.MapBacked(depthSize + plane, ReadWrite); + var stencilAddress = address + depthSize; + harness.Write(stencilAddress, Enumerable.Repeat((byte)0x5a, (int)plane).ToArray()); + + // A texture over the first half of the plane, cached before the depth target binds. + var texture = LinearRequest(stencilAddress, plane / 2, Format.R8Uint, GuestPixelFormat.Bits8UInt, GuestImageType.Color2D, new Extent3D(width, width / 2, 1), 1, 1, 1); + harness.Acquire(ref texture); + + var depth = LinearRequest(address, depthSize, Format.D32SfloatS8Uint, GuestPixelFormat.Bits32Float, GuestImageType.Color2D, new Extent3D(width, width, 1), 1, 4, 1); + depth.Description.TileMode = GuestTileMode.Depth; + depth = AsDepthTarget(depth, Format.D32SfloatS8Uint); + depth.Description.Stencil = new GuestSpan(stencilAddress, plane); + depth.View = depth.View with { Aspect = ImageAspectFlags.DepthBit | ImageAspectFlags.StencilBit }; + var depthId = harness.Acquire(ref depth); + Assert.All(harness.ReadImageBytes(harness.Image(depthId), ImageAspectFlags.StencilBit)[..(int)plane], value => Assert.Equal(0x5a, value)); + harness.Shutdown(); + } + + [Fact] + public void SmallerStencilStorageView_MovesOnlyItsRegion() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + var stencilFormat = SupportedStencilFormat(_vulkan, SampleCountFlags.Count1Bit); + if (stencilFormat == Format.Undefined) return; + using var harness = new CacheHarness(_vulkan); + var address = harness.MapBacked(0x10000, ReadWrite); + var depthAddress = address + 0xc000; + var stencilAddress = address + 0xd000; + var initial = Enumerable.Range(0x10, 16).Select(value => (byte)value).ToArray(); + harness.Write(stencilAddress, initial); + var depth = LinearRequest(depthAddress, 32, stencilFormat, GuestPixelFormat.Bits16UNorm, GuestImageType.Color2D, new Extent3D(8, 2, 1), 1, 2, 1); + depth = AsDepthTarget(depth, stencilFormat); + depth.Description.Stencil = new GuestSpan(stencilAddress, 16); + depth.View = depth.View with { Aspect = ImageAspectFlags.DepthBit | ImageAspectFlags.StencilBit }; + var depthId = harness.Acquire(ref depth); + var attachment = harness.Image(depthId); + var buffer = harness.Worker.Run(() => harness.Cache.GetUtilityBuffer(GpuBufferUsage.DeviceLocal)); + + // A 4x1 view of the 8x2 plane: the storage has the view's size and holds its texels. + var storage = harness.Worker.Run(() => + { + var image = attachment.CreateStencilStorageImage(4, 1); + attachment.CopyStencilStorage(image, buffer, writeBack: false); + return image; + }); + Assert.Equal(new Extent3D(4, 1, 1), storage.Backing.Extent); + Assert.Equal(initial[..4], harness.ReadImageBytes(storage)[..4]); + + // The plane changes everywhere; writing the storage back restores only its region. + harness.Write(stencilAddress, Enumerable.Repeat((byte)0xaa, 16).ToArray()); + Assert.True(harness.WriteFault(stencilAddress)); + Assert.Equal(depthId, harness.Acquire(ref depth)); + harness.Worker.Run(() => attachment.CopyStencilStorage(storage, buffer, writeBack: true)); + var plane = harness.ReadImageBytes(attachment, ImageAspectFlags.StencilBit)[..16]; + Assert.Equal(initial[..4], plane[..4]); + Assert.All(plane[4..], value => Assert.Equal(0xaa, value)); + harness.Worker.Run(storage.Dispose); + harness.Shutdown(); + } + + [Fact] + public void AlternatingStencilPlanes_WriteTheReleasedPlaneBack() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + var stencilFormat = SupportedStencilFormat(_vulkan, SampleCountFlags.Count1Bit); + if (stencilFormat == Format.Undefined) return; + using var harness = new CacheHarness(_vulkan); + var address = harness.MapBacked(0x10000, ReadWrite); + var depthAddress = address + 0xc000; + var firstPlane = address + 0xd000; + var secondPlane = address + 0xe000; + harness.Write(firstPlane, [0x11, 0x11, 0x11, 0x11]); + harness.Write(secondPlane, [0x22, 0x22, 0x22, 0x22]); + ImageRequest Bind(ulong plane) + { + var request = LinearRequest(depthAddress, 8, stencilFormat, GuestPixelFormat.Bits16UNorm, GuestImageType.Color2D, new Extent3D(4, 1, 1), 1, 2, 1); + request = AsDepthTarget(request, stencilFormat); + request.Description.Stencil = new GuestSpan(plane, 4); + request.View = request.View with { Aspect = ImageAspectFlags.DepthBit | ImageAspectFlags.StencilBit }; + return request; + } + + byte[] Stencil(ResourceSlotIdentifier id) => harness.ReadImageBytes(harness.Image(id), ImageAspectFlags.StencilBit)[..4]; + + var first = Bind(firstPlane); + var depthId = harness.Acquire(ref first); + // The GPU writes the first plane through the depth image. + Assert.True(harness.Worker.Run(() => harness.Images.TryClearImageFromBuffer(firstPlane, 4, 0x07070707))); + Assert.Equal(new byte[] { 0x07, 0x07, 0x07, 0x07 }, Stencil(depthId)); + + // The same depth image takes the second plane: the first one reaches guest memory. + var second = Bind(secondPlane); + Assert.Equal(depthId, harness.Acquire(ref second)); + Assert.Equal(new byte[] { 0x22, 0x22, 0x22, 0x22 }, Stencil(depthId)); + harness.Worker.Run(() => harness.Scheduler.Finish()); + Assert.Equal(new byte[] { 0x07, 0x07, 0x07, 0x07 }, harness.Read(firstPlane, 4)); + + // Back on the first plane, the depth image reloads what the GPU wrote. + first = Bind(firstPlane); + Assert.Equal(depthId, harness.Acquire(ref first)); + Assert.Equal(new byte[] { 0x07, 0x07, 0x07, 0x07 }, Stencil(depthId)); + harness.Shutdown(); + } + + // Another depth buffer with a larger stencil plane at the same address replaces the first + // one's association. A plane the GPU changed reaches guest memory first, and the new depth + // buffer loads it from there. + [Fact] + public void LargerStencilPlaneAtTheSameAddress_WritesTheReplacedPlaneBack() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + var stencilFormat = SupportedStencilFormat(_vulkan, SampleCountFlags.Count1Bit); + if (stencilFormat == Format.Undefined) return; + using var fatal = new SharpEmu.Libs.Tests.Gpu.Scheduling.FatalScope(); + using var harness = new CacheHarness(_vulkan); + var address = harness.MapBacked(0x10000, ReadWrite); + var plane = address + 0xd000; + harness.Write(plane, [0x11, 0x11, 0x11, 0x11, 0x22, 0x22, 0x22, 0x22]); + ImageRequest Bind(ulong depthAddress, uint width) + { + var request = LinearRequest(depthAddress, width * 2, stencilFormat, GuestPixelFormat.Bits16UNorm, GuestImageType.Color2D, new Extent3D(width, 1, 1), 1, 2, 1); + request = AsDepthTarget(request, stencilFormat); + request.Description.Stencil = new GuestSpan(plane, width); + request.View = request.View with { Aspect = ImageAspectFlags.DepthBit | ImageAspectFlags.StencilBit }; + return request; + } + + var small = Bind(address + 0xa000, 4); + harness.Acquire(ref small); + Assert.True(harness.Worker.Run(() => harness.Images.TryClearImageFromBuffer(plane, 4, 0x07070707))); + + var large = Bind(address + 0xb000, 8); + var largeId = harness.Acquire(ref large); + harness.Worker.Run(() => harness.Scheduler.Finish()); + Assert.Equal(new byte[] { 0x07, 0x07, 0x07, 0x07, 0x22, 0x22, 0x22, 0x22 }, harness.Read(plane, 8)); + Assert.Equal(new byte[] { 0x07, 0x07, 0x07, 0x07, 0x22, 0x22, 0x22, 0x22 }, + harness.ReadImageBytes(harness.Image(largeId), ImageAspectFlags.StencilBit)[..8]); + harness.Shutdown(); + } } diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Images/GuestImageCacheTests.Metadata.cs b/tests/SharpEmu.Libs.Tests/Gpu/Images/GuestImageCacheTests.Metadata.cs index 2eb5a41..ce3c27c 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Images/GuestImageCacheTests.Metadata.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Images/GuestImageCacheTests.Metadata.cs @@ -32,8 +32,12 @@ public sealed partial class GuestImageCacheTests Assert.True(harness.Images.IsMetadataCleared(entries[2], 0, out var fill)); Assert.Equal(0xffffffffu, fill); Assert.False(harness.Images.IsMetadataCleared(entries[0], 0)); - Assert.False(harness.Images.IsMetadataCleared(entries[2], 32)); - Assert.False(harness.Images.SetMetadataSlice(entries[2], 32, true)); + // A broad clear covers every array slice; the hardware's 2048-slice limit bounds tracking. + Assert.True(harness.Images.IsMetadataCleared(entries[2], 86)); + Assert.True(harness.Images.SetMetadataSlice(entries[2], 86, false)); + Assert.False(harness.Images.IsMetadataCleared(entries[2], 86)); + Assert.False(harness.Images.IsMetadataCleared(entries[2], 2048)); + Assert.False(harness.Images.SetMetadataSlice(entries[2], 2048, true)); Assert.False(harness.Images.IsMetadata(address + 0x3000)); Assert.False(harness.Images.ClearMetadata(address + 0x3000)); harness.Shutdown(); diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Images/GuestImageCacheTests.Ownership.cs b/tests/SharpEmu.Libs.Tests/Gpu/Images/GuestImageCacheTests.Ownership.cs index 21492f8..bda0b6d 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Images/GuestImageCacheTests.Ownership.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Images/GuestImageCacheTests.Ownership.cs @@ -7,6 +7,7 @@ using SharpEmu.HLE.Host; using SharpEmu.Libs.Gpu.Buffers; using SharpEmu.Libs.Gpu.Images; using SharpEmu.Libs.Tests.Gpu.Buffers; +using SharpEmu.Libs.Tests.Gpu.Scheduling; using Silk.NET.Vulkan; using Xunit; using static SharpEmu.Libs.Tests.Gpu.Images.ImageCacheTestSupport; @@ -57,11 +58,90 @@ public sealed partial class GuestImageCacheTests Assert.True(harness.Images.Contains(exactId)); Assert.True(harness.Image(exactId).IsGpuModified); Assert.False(harness.Image(exactFloatId).IsBufferModified); - Assert.False(harness.Image(exactFloatId).IsGpuModified); + // The float alias copies the GPU-owned contents of the same-layout integer image. + Assert.True(harness.Image(exactFloatId).IsGpuModified); Assert.Equal(Bytes(0x3f234567u), harness.ReadImageBytes(harness.Image(exactFloatId))); harness.Shutdown(); } + // One surface read through two formats of the same layout is two cache images. The one + // used next copies the newer GPU contents of the other, which guest memory does not hold. + [Fact] + public void SameLayoutAlias_ReadsTheNewerGpuContentsOfTheOtherFormat() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var harness = new CacheHarness(_vulkan); + var address = harness.MapBacked(0x10000, ReadWrite); + harness.Write(address + 0x4000, Bytes(0x11111111u)); + var integer = Color32(address + 0x4000); + var integerId = harness.Acquire(ref integer, exactFormat: true); + Assert.True(harness.Worker.Run(() => harness.Images.TryClearImageFromBuffer(address + 0x4000, 4, 0xaabbccddu))); + + var floating = LinearRequest(address + 0x4000, 4, Format.R32Sfloat, GuestPixelFormat.Bits32Float, GuestImageType.Color2D, new Extent3D(1, 1, 1), 1, 4, 1); + var floatingId = harness.Acquire(ref floating, exactFormat: true); + + Assert.NotEqual(integerId, floatingId); + Assert.Equal(Bytes(0xaabbccddu), harness.ReadImageBytes(harness.Image(floatingId))); + Assert.Equal(Bytes(0x11111111u), harness.Read(address + 0x4000, 4)); + harness.Shutdown(); + } + + // An alias already holding the newest contents is not copied again. + [Fact] + public void SameLayoutAlias_DoesNotCopyContentsItAlreadyHolds() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var harness = new CacheHarness(_vulkan); + var address = harness.MapBacked(0x10000, ReadWrite); + harness.Write(address + 0x4000, Bytes(0x11111111u)); + var integer = Color32(address + 0x4000); + var integerId = harness.Acquire(ref integer, exactFormat: true); + Assert.True(harness.Worker.Run(() => harness.Images.TryClearImageFromBuffer(address + 0x4000, 4, 0xaabbccddu))); + var floating = LinearRequest(address + 0x4000, 4, Format.R32Sfloat, GuestPixelFormat.Bits32Float, GuestImageType.Color2D, new Extent3D(1, 1, 1), 1, 4, 1); + var floatingId = harness.Acquire(ref floating, exactFormat: true); + var floatingContents = harness.Image(floatingId).ContentSequence; + var integerContents = harness.Image(integerId).ContentSequence; + + var again = integer; + Assert.Equal(integerId, harness.Acquire(ref again, exactFormat: true)); + var floatingAgain = floating; + Assert.Equal(floatingId, harness.Acquire(ref floatingAgain, exactFormat: true)); + + Assert.Equal(integerContents, harness.Image(integerId).ContentSequence); + Assert.Equal(floatingContents, harness.Image(floatingId).ContentSequence); + Assert.Equal(integerContents, floatingContents); + harness.Shutdown(); + } + + // A formatted shader binding can cover only one cache page of a larger GPU image. + // Publishing the untouched image bytes must keep writes through that binding visible. + [Fact] + public void PartialFormattedWriteToGpuImage_ReachesTheCurrentBufferOwner() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var harness = new CacheHarness(_vulkan); + using var fatal = new FatalScope(); + var address = harness.MapBacked(0x10000, ReadWrite); + var request = Color32(address, 8192); + var imageId = harness.Acquire(ref request); + Assert.True(harness.Worker.Run(() => harness.Images.TryClearImageFromBuffer(address, 0x8000, 0x11223344))); + Assert.True(harness.Image(imageId).IsGpuModified); + + var writeAddress = address + 0x4000; + harness.Worker.Run(() => + { + var (bound, offset) = harness.Cache.ObtainBuffer(writeAddress, 4, isWritten: true, isTexelBuffer: true); + harness.Images.InvalidateMemoryFromGpu(writeAddress, 4); + bound.Fill(offset, 4, 0x55667788); + }); + + Assert.True(harness.Cache.DownloadToCpu(writeAddress, 4)); + Assert.Equal(0x55667788u, harness.ReadUInt32(writeAddress)); + Assert.True(harness.Cache.DownloadToCpu(address, 4)); + Assert.Equal(0x11223344u, harness.ReadUInt32(address)); + harness.Shutdown(); + } + [Fact] public void BufferWriteThenImageClear_TransfersOwnershipToTheImage() { @@ -89,6 +169,30 @@ public sealed partial class GuestImageCacheTests harness.Shutdown(); } + [Fact] + public void RawBufferWrite_InvalidatesMemoryCopiesAndKeepsGpuOwnedImages() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var harness = new CacheHarness(_vulkan); + var address = harness.MapBacked(0x10000, ReadWrite); + harness.Write(address, Bytes(0x01010101u)); + harness.Write(address + 0x6000, Bytes(0x02020202u)); + var copy = LinearRequest(address, 4, Format.R8G8B8A8Unorm, GuestPixelFormat.Bits8_8_8_8UNorm, GuestImageType.Color2D, new Extent3D(1, 1, 1), 1, 4, 1); + var copyId = harness.Find(ref copy); + var owned = LinearRequest(address + 0x6000, 4, Format.R8G8B8A8Unorm, GuestPixelFormat.Bits8_8_8_8UNorm, GuestImageType.Color2D, new Extent3D(1, 1, 1), 1, 4, 1); + var ownedId = harness.Find(ref owned); + Assert.True(harness.Worker.Run(() => harness.Images.TryClearImageFromBuffer(address + 0x6000, 4, 0xaabbccddu))); + Assert.True(harness.Image(ownedId).IsGpuModified); + + harness.Worker.Run(() => harness.Images.InvalidateMemoryCopiesFromGpu(address, 0x10000)); + + Assert.True(harness.Image(copyId).IsBufferModified); + Assert.False(harness.Image(ownedId).IsBufferModified); + Assert.True(harness.Image(ownedId).IsGpuModified); + Assert.Equal(new byte[] { 0xdd, 0xcc, 0xbb, 0xaa }, harness.ReadImageBytes(harness.Image(ownedId))); + harness.Shutdown(); + } + [Fact] public void FullColorClear_SkipsSourceBufferAndRetainsWriteObservation() { diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Images/GuestImageCacheTests.Readback.cs b/tests/SharpEmu.Libs.Tests/Gpu/Images/GuestImageCacheTests.Readback.cs index bd08b3b..3715c4e 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Images/GuestImageCacheTests.Readback.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Images/GuestImageCacheTests.Readback.cs @@ -14,6 +14,518 @@ namespace SharpEmu.Libs.Tests.Gpu.Images; // Readback publication: scheduled readbacks, garbage collection under pressure, depth planes. public sealed partial class GuestImageCacheTests { + [Theory] + [InlineData(1u)] + [InlineData(2u)] + public void LinearColorTarget_UsesExplicitRowPitch(uint channels) + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var harness = new CacheHarness(_vulkan); + const uint width = 37, height = 3, layers = 2; + var size = width * height * layers * channels; + var address = harness.MapBacked(0x4000, ReadWrite); + var input = Enumerable.Range(0, (int)size).Select(index => (byte)(index * 19 + 7)).ToArray(); + harness.Write(address, input); + var request = AsColorTarget(LinearRequest(address, size, + channels == 1 ? Format.R8Unorm : Format.R8G8Unorm, + channels == 1 ? GuestPixelFormat.Bits8UNorm : GuestPixelFormat.Bits8_8UNorm, + GuestImageType.Color2D, new Extent3D(width, height, 1), layers, channels, 1)); + var identifier = harness.Acquire(ref request); + Assert.Equal(input, harness.ReadImageBytes(harness.Image(identifier))); + harness.MarkGpuWritten(identifier); + var result = harness.Worker.Run(() => + { + using var output = new GpuBuffer(harness.Vulkan.DeviceInfo, harness.Scheduler, + GpuBufferUsage.DeviceLocal, address, GpuBuffer.AllFlags, (size + 3UL) & ~3UL); + Assert.True(harness.Images.TrySynchronizeBufferFromImage(output, address, size)); + return harness.CopyFromDevice(output.Handle, 0, size); + }); + Assert.Equal(input, result); + } + + [Theory] + [InlineData(1u)] + [InlineData(2u)] + public void ExpandedSrgbBacking_TiledReadbackPreservesInactiveBytes(uint channels) + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var harness = new CacheHarness(_vulkan); + const uint width = 37, height = 3, layers = 2; + const ulong size = 0x20000; + var address = harness.MapBacked(0x40000, ReadWrite); + harness.Write(address, Enumerable.Repeat((byte)0x11, (int)size).ToArray()); + var request = LinearRequest(address, size, Format.R8G8B8A8Srgb, + channels == 1 ? GuestPixelFormat.Bits8Srgb : GuestPixelFormat.Bits8_8Srgb, + GuestImageType.Color2D, new Extent3D(width, height, 1), layers, channels, 1); + request.Description.TileMode = GuestTileMode.RenderTarget; + var identifier = harness.Acquire(ref request); + var wide = harness.ReadImageBytes(harness.Image(identifier)); + for (var pixel = 0; pixel < width * height * layers; pixel++) + for (var channel = 0; channel < 4; channel++) + Assert.Equal(channel < channels ? (byte)0x11 : channel == 3 ? (byte)255 : (byte)0, + wide[pixel * 4 + channel]); + harness.MarkGpuWritten(identifier); + var result = harness.Worker.Run(() => + { + using var output = new GpuBuffer(harness.Vulkan.DeviceInfo, harness.Scheduler, + GpuBufferUsage.DeviceLocal, address, GpuBuffer.AllFlags, size); + output.Fill(0, size, 0x5a5a5a5a); + Assert.True(harness.Images.TrySynchronizeBufferFromImage(output, address, size)); + return harness.CopyFromDevice(output.Handle, 0, size); + }); + Assert.Equal((int)(width * height * layers * channels), result.Count(value => value == 0x11)); + Assert.Equal(result.Length - (int)(width * height * layers * channels), result.Count(value => value == 0x5a)); + } + + [Theory] + [InlineData(1u, 256u)] + [InlineData(2u, 256u)] + [InlineData(1u, 37u)] + [InlineData(2u, 37u)] + public void ExpandedSrgbBacking_RoundTripsEncodedGuestChannels(uint channels, uint width) + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var harness = new CacheHarness(_vulkan); + var address = harness.MapBacked(0x4000, ReadWrite); + const uint height = 3, layers = 2; + var pitch = (width * channels + 255) / 256 * 256 / channels; + var size = pitch * height * layers * channels; + var input = Enumerable.Range(0, (int)size).Select(index => (byte)(index * 37 + 11)).ToArray(); + harness.Write(address, input); + var request = LinearRequest(address, size, Format.R8G8B8A8Srgb, + channels == 1 ? GuestPixelFormat.Bits8Srgb : GuestPixelFormat.Bits8_8Srgb, + GuestImageType.Color2D, new Extent3D(width, height, 1), layers, channels, 1); + request.Description.Pitch = pitch; + request.Description.MipLayout[0].Pitch = pitch; + var identifier = harness.Acquire(ref request); + var expectedWide = new byte[width * height * layers * 4]; + for (var pixel = 0; pixel < width * height * layers; pixel++) + { + expectedWide[pixel * 4 + 3] = 255; + for (var channel = 0; channel < channels; channel++) + expectedWide[pixel * 4 + channel] = input[(pixel / width * pitch + pixel % width) * channels + channel]; + } + Assert.Equal(expectedWide, harness.ReadImageBytes(harness.Image(identifier))); + harness.MarkGpuWritten(identifier); + harness.Worker.Run(() => + { + using var output = new GpuBuffer(harness.Vulkan.DeviceInfo, harness.Scheduler, + GpuBufferUsage.DeviceLocal, address, GpuBuffer.AllFlags, size); + output.Fill(0, size, 0x5a5a5a5a); + Assert.True(harness.Images.TrySynchronizeBufferFromImage(output, address, size)); + var expectedGuest = Enumerable.Repeat((byte)0x5a, (int)size).ToArray(); + for (var row = 0; row < height * layers; row++) + input.AsSpan((int)(row * pitch * channels), (int)(width * channels)) + .CopyTo(expectedGuest.AsSpan((int)(row * pitch * channels))); + Assert.Equal(expectedGuest, harness.CopyFromDevice(output.Handle, 0, size)); + }); + } + + [Fact] + public void GarbageCollector_KeepsGpuContentsWhenBufferOverlapBlocksReadback() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var harness = new CacheHarness(_vulkan); + var address = harness.MapBacked(0x10000, ReadWrite); + harness.Write(address, Bytes(0x12345678u)); + var request = Color32(address); + var identifier = harness.Acquire(ref request); + harness.Worker.Run(() => + { + Assert.NotNull(harness.Cache.ObtainBuffer(address, 4, isWritten: true).Buffer); + }); + harness.MarkGpuWritten(identifier); + Assert.True(harness.Cache.HasGpuDirtyBytes(address, 4)); + Assert.True(harness.Image(identifier).SafeToDownload); + harness.Worker.Run(() => + { + harness.Images.SetCollectionThresholds(0, 0, ulong.MaxValue, 81); + harness.Images.ResetRecency(new[] { identifier }, 81); + harness.Images.RunGarbageCollector(); + }); + Assert.True(harness.Images.Contains(identifier)); + Assert.True(harness.Image(identifier).IsGpuModified); + harness.Shutdown(); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void GarbageCollector_ResolvesSamePageWritesBeforeDiscardingGpuContents(bool imageBytesChanged) + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var harness = new CacheHarness(_vulkan); + var address = harness.MapBacked(0x10000, ReadWrite); + var imageAddress = address + 0x8000; + harness.Write(imageAddress, Bytes(0x12345678u)); + var request = Color32(imageAddress); + var identifier = harness.Acquire(ref request); + harness.MarkGpuWritten(identifier); + Assert.True(harness.WriteFault(address + 0x8080)); + Assert.True(harness.Image(identifier).IsMaybeCpuDirty); + Assert.False(harness.Image(identifier).NeedsMaybeCpuHash); + if (imageBytesChanged) + harness.Write(imageAddress, Bytes(0x87654321u)); + + harness.Worker.Run(() => + { + harness.Images.SetCollectionThresholds(0, ulong.MaxValue, ulong.MaxValue, 17); + harness.Images.ResetRecency(new[] { identifier }, 17); + harness.Images.RunGarbageCollector(); + }); + + Assert.Equal(!imageBytesChanged, harness.Images.Contains(identifier)); + if (!imageBytesChanged) + { + Assert.True(harness.Image(identifier).IsGpuModified); + Assert.False(harness.Image(identifier).IsCpuDirty); + Assert.Equal(Bytes(0x12345678u), harness.ReadImageBytes(harness.Image(identifier))); + } + harness.Shutdown(); + } + + + [Fact] + public void AllocationPressure_CompletesRetirementAndKeepsCurrentImages() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var fatal = new SharpEmu.Libs.Tests.Gpu.Scheduling.FatalScope(); + using var harness = new CacheHarness(_vulkan); + var address = harness.MapBacked(0x10000, ReadWrite); + harness.Images.SetCollectionThresholds(ulong.MaxValue, ulong.MaxValue, ulong.MaxValue, 1); + var oldRequest = Color32(address); + var oldImage = harness.Acquire(ref oldRequest); + var oldOwner = harness.Image(oldImage); + harness.Images.SetCollectionThresholds(ulong.MaxValue, ulong.MaxValue, ulong.MaxValue, 2); + var currentRequest = Color32(address + 256); + var currentImage = harness.Acquire(ref currentRequest); + var pool = (OptimalImageMemoryPool)typeof(GuestImageCache).GetField("_imageMemoryPool", + System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.NonPublic)!.GetValue(harness.Images)!; + var before = harness.Scheduler.CurrentTick; + var callbackRan = false; + harness.Worker.Run(() => + { + harness.Scheduler.QueueCompletionAction(() => + { + // Completion must be able to acquire the image cache lock. + _ = harness.Images.FindImagesInRangeForTest(address + 256, 256, false); + callbackRan = true; + }); + harness.Images.SetCollectionThresholds(0, 0, pool.AllocatedBytes - 1, 2); + var nextRequest = Color32(address + 512); + _ = harness.Images.FindImage(ref nextRequest); + }); + Assert.True(callbackRan); + Assert.True(harness.Scheduler.CurrentTick > before); + Assert.False(harness.Images.Contains(oldImage)); + Assert.False(oldOwner.Backing.Exists); + Assert.True(harness.Images.Contains(currentImage)); + harness.Shutdown(); + } + + [Fact] + public void AllocationPressure_DoesNotRetireTheEntireOldWorkingSetBeforeCompletion() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var harness = new CacheHarness(_vulkan); + var address = harness.MapBacked(0x10000, ReadWrite); + harness.Images.SetCollectionThresholds(ulong.MaxValue, ulong.MaxValue, ulong.MaxValue, 1); + var oldImages = new List(); + for (var i = 0; i < 60; i++) + { + var request = Color32(address + (ulong)i * 256); + oldImages.Add(harness.Acquire(ref request)); + } + var pool = (OptimalImageMemoryPool)typeof(GuestImageCache).GetField("_imageMemoryPool", + System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.NonPublic)!.GetValue(harness.Images)!; + harness.Worker.Run(() => + { + harness.Images.SetCollectionThresholds(0, 0, pool.AllocatedBytes - 1, 2); + var request = Color32(address + 60 * 256); + _ = harness.Images.FindImage(ref request); + }); + // One retirement batch satisfies this small request. Waiting for the + // total resident counter to drop instead used to discard all 60. + Assert.InRange(oldImages.Count(harness.Images.Contains), 20, 59); + harness.Shutdown(); + } + + [Fact] + public void OversizedReadbacks_PublishBothImagesWithoutGrowingTheSharedRing() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var fatal = new SharpEmu.Libs.Tests.Gpu.Scheduling.FatalScope(); + const uint width = 4096, height = 2049; + const ulong size = (ulong)width * height * 4; + using var harness = new CacheHarness(_vulkan, backingBytes: 80UL * 1024 * 1024); + var address = harness.MapBacked(size * 2, ReadWrite); + var download = harness.Cache.GetUtilityBuffer(GpuBufferUsage.Download); + var ringSize = download.Size; + Assert.True(size > ringSize); + ulong[] samples = [0, size / 2, size - 4]; + uint[] values = [0x11223344, 0x55667788]; + var images = new List(); + var tick = harness.Scheduler.CurrentTick; + for (var index = 0; index < values.Length; index++) + { + var imageAddress = address + (ulong)index * size; + var request = LinearRequest(imageAddress, size, Format.R8G8B8A8Unorm, + GuestPixelFormat.Bits8_8_8_8UNorm, GuestImageType.Color2D, new Extent3D(width, height, 1), 1, 4, 1); + var image = harness.Find(ref request); + var value = values[index]; + harness.Worker.Run(() => Assert.True(harness.Images.TryClearImageFromBuffer(imageAddress, size, value))); + foreach (var sample in samples) harness.Write(imageAddress + sample, Bytes(0u)); + images.Add(image); + } + var baselineAllocations = harness.Vulkan.DeviceInfo.LiveAllocations; + foreach (var image in images) + Assert.True(harness.Worker.Run(() => harness.Images.TryDownloadForTest(image))); + Assert.Equal(tick, harness.Scheduler.CurrentTick); + foreach (var index in Enumerable.Range(0, values.Length)) + foreach (var sample in samples) Assert.Equal(0u, harness.ReadUInt32(address + (ulong)index * size + sample)); + harness.Finish(); + Assert.Equal(baselineAllocations, harness.Vulkan.DeviceInfo.LiveAllocations); + foreach (var index in Enumerable.Range(0, values.Length)) + foreach (var sample in samples) Assert.Equal(values[index], harness.ReadUInt32(address + (ulong)index * size + sample)); + Assert.Same(download, harness.Cache.GetUtilityBuffer(GpuBufferUsage.Download)); + Assert.Equal(ringSize, download.Size); + harness.Shutdown(); + } + + [Fact] + public void Pressure_RetiresTiledGpuImageAndPreservesItsGuestContents() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var harness = new CacheHarness(_vulkan); + const ulong size = 0x10000; + const uint clear = 0x3f200000, stale = 0xdeadbeef; + var address = harness.MapBacked(size, ReadWrite); + harness.Write(address, Bytes(Enumerable.Repeat(stale, (int)(size / 4)).ToArray())); + var request = AsDepthTarget(LinearRequest(address, size, Format.D32Sfloat, + GuestPixelFormat.Bits32Float, GuestImageType.Color2D, new Extent3D(3, 2, 1), 1, 4, 1), Format.D32Sfloat); + request.Description.Pitch = 4; + request.Description.TileMode = GuestTileMode.Depth; + request.Description.MipLayout[0] = new MipLevelLayout { Offset = 0, Size = size, Pitch = 4, Height = 2 }; + var image = harness.Find(ref request); + harness.Worker.Run(() => + { + Assert.True(harness.Images.TryClearImageFromBuffer(address, size, clear)); + harness.Images.SetCollectionThresholds(0, 0, 1, 2); + harness.Images.ResetRecency([image], 2); + harness.Images.RunGarbageCollector(); + }); + Assert.False(harness.Images.Contains(image)); + Assert.All(harness.Read(address, (int)size).Chunk(4), bytes => Assert.Equal(stale, BitConverter.ToUInt32(bytes))); + harness.Finish(); + var result = harness.Read(address, (int)size).Chunk(4).Select(bytes => BitConverter.ToUInt32(bytes)).ToArray(); + Assert.Equal(6, result.Count(value => value == clear)); + Assert.Equal(result.Length - 6, result.Count(value => value == stale)); + harness.Shutdown(); + } + + // A tiled image retired under pressure is published only when its readback completes. Asked + // for again before then, the image must not be rebuilt from the guest bytes it supersedes. + [Fact] + public void Pressure_TiledImageFoundAgainBeforeItsReadbackCompletes_KeepsItsGpuContents() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var harness = new CacheHarness(_vulkan); + const ulong size = 0x10000; + const uint clear = 0x3f200000, stale = 0xdeadbeef; + var address = harness.MapBacked(size, ReadWrite); + harness.Write(address, Bytes(Enumerable.Repeat(stale, (int)(size / 4)).ToArray())); + ImageRequest Request() + { + var request = AsDepthTarget(LinearRequest(address, size, Format.D32Sfloat, + GuestPixelFormat.Bits32Float, GuestImageType.Color2D, new Extent3D(3, 2, 1), 1, 4, 1), Format.D32Sfloat); + request.Description.Pitch = 4; + request.Description.TileMode = GuestTileMode.Depth; + request.Description.MipLayout[0] = new MipLevelLayout { Offset = 0, Size = size, Pitch = 4, Height = 2 }; + return request; + } + + var first = Request(); + var image = harness.Find(ref first); + harness.Worker.Run(() => + { + Assert.True(harness.Images.TryClearImageFromBuffer(address, size, clear)); + harness.Images.SetCollectionThresholds(0, 0, 1, 2); + harness.Images.ResetRecency([image], 2); + harness.Images.RunGarbageCollector(); + Assert.False(harness.Images.Contains(image)); + harness.Images.SetCollectionThresholds(ulong.MaxValue, ulong.MaxValue, ulong.MaxValue, 3); + }); + var second = Request(); + var again = harness.Acquire(ref second); + + // Publish the second image's contents the same way and compare with the GPU's value. + harness.Finish(); + harness.MarkGpuWritten(again); + harness.Worker.Run(() => + { + harness.Images.SetCollectionThresholds(0, 0, 1, 4); + harness.Images.ResetRecency([again], 4); + harness.Images.RunGarbageCollector(); + }); + harness.Finish(); + var result = harness.Read(address, (int)size).Chunk(4).Select(bytes => BitConverter.ToUInt32(bytes)).ToArray(); + Assert.Equal(6, result.Count(value => value == clear)); + harness.Shutdown(); + } + + [Fact] + public void Pressure_CompletesTiledReadbackBatchesBeforeAllocatingMoreScratch() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var harness = new CacheHarness(_vulkan); + const ulong size = 0x10000; + const uint clear = 0x3f200000, stale = 0xdeadbeef; + var address = harness.MapBacked(3 * size, ReadWrite); + harness.Write(address, Bytes(Enumerable.Repeat(stale, (int)(3 * size / 4)).ToArray())); + harness.Images.SetCollectionThresholds(ulong.MaxValue, ulong.MaxValue, ulong.MaxValue, 1); + var images = new List(); + for (var index = 0; index < 3; index++) + { + var imageAddress = address + (ulong)index * size; + var request = AsDepthTarget(LinearRequest(imageAddress, size, Format.D32Sfloat, + GuestPixelFormat.Bits32Float, GuestImageType.Color2D, new Extent3D(3, 2, 1), 1, 4, 1), Format.D32Sfloat); + request.Description.Pitch = 4; + request.Description.TileMode = GuestTileMode.Depth; + request.Description.MipLayout[0] = new MipLevelLayout { Offset = 0, Size = size, Pitch = 4, Height = 2 }; + images.Add(harness.Find(ref request)); + harness.Worker.Run(() => Assert.True(harness.Images.TryClearImageFromBuffer(imageAddress, size, clear))); + } + harness.Worker.Run(() => + { + harness.Images.TilerForTest.OutstandingScratchBudget = 1; + harness.Images.SetCollectionThresholds(0, 0, ulong.MaxValue, 2); + harness.Images.ResetRecency(System.Runtime.InteropServices.CollectionsMarshal.AsSpan(images), 2); + var tick = harness.Scheduler.CurrentTick; + harness.Images.RunGarbageCollector(); + Assert.True(harness.Scheduler.CurrentTick > tick); + Assert.All(images, image => Assert.False(harness.Images.Contains(image))); + Assert.Equal(0UL, harness.Images.TilerForTest.PooledScratchBytes); + }); + // Earlier batches have already published; the final batch may be pending. + Assert.Contains(clear, harness.Read(address, (int)size).Chunk(4).Select(bytes => BitConverter.ToUInt32(bytes))); + harness.Finish(); + for (var index = 0; index < 3; index++) + { + var words = harness.Read(address + (ulong)index * size, (int)size).Chunk(4).Select(bytes => BitConverter.ToUInt32(bytes)).ToArray(); + Assert.Equal(6, words.Count(word => word == clear)); + Assert.Equal(words.Length - 6, words.Count(word => word == stale)); + } + harness.Shutdown(); + } + + [Fact] + public void ProtectedStencilProxies_DoNotHideLaterEvictableImages() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var harness = new CacheHarness(_vulkan); + var address = harness.MapBacked(0x10000, ReadWrite); + harness.Images.SetCollectionThresholds(ulong.MaxValue, ulong.MaxValue, ulong.MaxValue, 1); + var ownerRequest = Color32(address); + var owner = harness.Acquire(ref ownerRequest); + var proxies = new List(); + for (var index = 1; index <= 40; index++) + { + var request = Color32(address + (ulong)index * 256); + var proxy = harness.Acquire(ref request); + harness.Image(proxy).DepthOwner = owner; + proxies.Add(proxy); + } + var victimRequest = Color32(address + 41 * 256); + var victim = harness.Acquire(ref victimRequest); + harness.Worker.Run(() => + { + harness.Images.SetCollectionThresholds(0, 0, 1, 2); + _ = harness.Images.GetImage(owner); + harness.Images.RunGarbageCollector(); + }); + Assert.False(harness.Images.Contains(victim)); + Assert.True(harness.Images.Contains(owner)); + Assert.All(proxies, proxy => Assert.True(harness.Images.Contains(proxy))); + harness.Finish(); + harness.Shutdown(); + } + + [Fact] + public void CriticalPressure_RetiresRecentlyUnusedImagesButKeepsCurrentBindings() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var harness = new CacheHarness(_vulkan); + var address = harness.MapBacked(0x10000, ReadWrite); + var oldRequest = Color32(address); + var currentRequest = Color32(address + 256); + harness.Images.SetCollectionThresholds(ulong.MaxValue, ulong.MaxValue, ulong.MaxValue, 1); + var oldImage = harness.Acquire(ref oldRequest); + harness.Images.SetCollectionThresholds(ulong.MaxValue, ulong.MaxValue, ulong.MaxValue, 2); + var currentImage = harness.Acquire(ref currentRequest); + harness.Worker.Run(() => + { + harness.Images.SetCollectionThresholds(0, 0, 1, 2); + harness.Images.RunGarbageCollector(); + }); + Assert.False(harness.Images.Contains(oldImage)); + Assert.True(harness.Images.Contains(currentImage)); + // Retirement must not destroy memory still referenced by recorded work. + Assert.True(harness.Image(oldImage).Backing.Exists); + harness.Finish(); + harness.Shutdown(); + } + + [Fact] + public void SmallTextures_ShareDeviceMemoryWithoutSharingTheirContents() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var harness = new CacheHarness(_vulkan); + var address = harness.MapBacked(0x10000, ReadWrite); + var memories = new HashSet(); + for (uint index = 0; index < 64; index++) + { + var request = Color32(address + index * 256); + harness.Write(request.Description.Data.Address, Bytes(index + 1)); + var id = harness.Acquire(ref request); + memories.Add(harness.Image(id).Backing.Memory.Handle); + var textureAddress = request.Description.Data.Address; + var clearValue = index + 128; + harness.Worker.Run(() => + { + Assert.True(harness.Images.TryClearImageFromBuffer(textureAddress, 4, clearValue)); + Assert.True(harness.Images.TryDownloadForTest(id)); + }); + } + Assert.Single(memories); + harness.Finish(); + for (uint index = 0; index < 64; index++) + Assert.Equal(index + 128, BitConverter.ToUInt32(harness.Read(address + index * 256, 4))); + harness.Shutdown(); + } + + [Fact] + public void RetiredImages_StayAccountedUntilTheirGpuCompletion() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var harness = new CacheHarness(_vulkan); + var address = harness.MapBacked(0x10000, ReadWrite); + var request = Color32(address); + var id = harness.Acquire(ref request); + var allocated = harness.Images.TotalUsedMemory; + Assert.True(allocated > 0); + harness.Images.SetCollectionThresholds(allocated, allocated, allocated, 0); + Assert.False(harness.Images.RetirementOverBudget); + harness.Worker.Run(() => ((IGuestImageStore)harness.Images).Unregister(address, 4)); + Assert.False(harness.Images.Contains(id)); + Assert.True(harness.Image(id).Backing.Exists); + Assert.Equal(allocated, harness.Images.TotalUsedMemory); + Assert.True(harness.Images.RetirementOverBudget); + harness.Finish(); + Assert.False(harness.Images.RetirementOverBudget); + Assert.Equal(0UL, harness.Images.TotalUsedMemory); + harness.Shutdown(); + } + [Fact] public void GarbageCollector_RetiresUnderPressureAndPublishesInOneTick() { @@ -125,6 +637,52 @@ public sealed partial class GuestImageCacheTests harness.Shutdown(); } + // Collection runs before every draw while memory is under pressure. Below the critical + // level it must leave the previous frame's images alone: the rest of the frame samples them + // again, so evicting them there recreated and uploaded thousands of images every frame. + [Fact] + public void GarbageCollector_MidFramePressureKeepsThePreviousFramesImages() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var harness = new CacheHarness(_vulkan); + var address = harness.MapBacked(0x400000, ReadWrite); + var request = Color32(address + 0x330000); + ResourceSlotIdentifier[] images = [harness.Find(ref request)]; + harness.Worker.Run(() => + { + harness.Images.SetCollectionThresholds(0, 0, ulong.MaxValue, 1); + harness.Images.ResetRecency(images, 1); + for (var draw = 0; draw < 64; draw++) + { + harness.Images.RunGarbageCollector(endsFrame: false); + } + }); + Assert.True(harness.Images.Contains(images[0])); + + harness.Worker.Run(() => harness.Images.RunGarbageCollector(endsFrame: true)); + Assert.False(harness.Images.Contains(images[0])); + harness.Shutdown(); + } + + // At the critical level an allocation can fail, so the mid-frame pass still reclaims. + [Fact] + public void GarbageCollector_MidFrameCriticalPressureReclaimsThePreviousFramesImages() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var harness = new CacheHarness(_vulkan); + var address = harness.MapBacked(0x400000, ReadWrite); + var request = Color32(address + 0x330000); + ResourceSlotIdentifier[] images = [harness.Find(ref request)]; + harness.Worker.Run(() => + { + harness.Images.SetCollectionThresholds(0, 0, 0, 1); + harness.Images.ResetRecency(images, 1); + harness.Images.RunGarbageCollector(endsFrame: false); + }); + Assert.False(harness.Images.Contains(images[0])); + harness.Shutdown(); + } + [Fact] public void ScheduledReadback_PublishesAfterTheTickAndKeepsTheImage() { @@ -464,4 +1022,130 @@ public sealed partial class GuestImageCacheTests harness.Shutdown(); } + + [Fact] + public void LinearReadback_LargerThanItsGuestRange_IsNotCopied() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var fatal = new SharpEmu.Libs.Tests.Gpu.Scheduling.FatalScope(); + const uint width = 516, height = 516; + const ulong size = (ulong)width * height; + var layout = TextureTransferLayout.Compute(GuestPixelFormat.Bits8UNorm, width, height, 1, 1, GuestTileMode.Linear, size, false, false, "test"); + // The padded rows of the format's linear layout reach past the bytes the guest declared. + Assert.True(TextureTransferLayout.CopyFootprint(layout.BuildCopies(), new TileElementLayout(1, 1, 1)) > size); + + using var harness = new CacheHarness(_vulkan); + var address = harness.MapBacked(0x100000, ReadWrite); + var request = LinearRequest(address, size, Format.R8Unorm, GuestPixelFormat.Bits8UNorm, GuestImageType.Color2D, + new Extent3D(width, height, 1), 1, 1, 1); + // Like a texture, the description carries no level layout of its own. + request.Description.MipLayout[0] = new MipLevelLayout { Pitch = 1, Height = 1 }; + harness.Write(address, Bytes(0x01020304u)); + var image = harness.Acquire(ref request); + // A GPU-modified image is the newest copy of its range, so it is eligible for readback. + harness.Image(image).MarkGpuModified(); + Assert.True(harness.Image(image).SafeToDownload); + + // Copying that layout would write past the readback storage, so nothing is read back. + Assert.False(harness.Worker.Run(() => harness.Images.TryDownloadForTest(image))); + harness.Finish(); + Assert.Equal(0x01020304u, harness.ReadUInt32(address)); + harness.Shutdown(); + } + + // A linear render target's rows follow the guest's pitch, which need not be aligned like a + // texture's: its contents go to the GPU and come back without shifting a row. + [Fact] + public void LinearTarget_WithAnUnalignedGuestPitch_RoundTripsItsRows() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + const uint width = 516, height = 516; + const ulong size = (ulong)width * height; + using var harness = new CacheHarness(_vulkan); + var address = harness.MapBacked(0x100000, ReadWrite); + var contents = new byte[size]; + for (var index = 0; index < contents.Length; index++) + { + contents[index] = (byte)(index * 7 + index / width); + } + + harness.Write(address, contents); + var request = LinearRequest(address, size, Format.R8Unorm, GuestPixelFormat.Bits8UNorm, GuestImageType.Color2D, + new Extent3D(width, height, 1), 1, 1, 1); + var image = harness.Acquire(ref request); + harness.Image(image).MarkGpuModified(); + + Assert.True(harness.Worker.Run(() => harness.Images.TryDownloadForTest(image))); + harness.Finish(); + Assert.Equal(contents, harness.Read(address, (int)size)); + harness.Shutdown(); + } + + // The image budget is an estimate, so the device can still run out of memory. The cache + // frees what it can and tries the allocation once more instead of stopping. + [Fact] + public void ImageAllocation_RetriesOnceAfterTheDeviceRunsOutOfMemory() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var fatal = new SharpEmu.Libs.Tests.Gpu.Scheduling.FatalScope(); + const uint width = 4096, height = 4096; + const ulong size = (ulong)width * height * 4; + using var harness = new CacheHarness(_vulkan, backingBytes: 80UL * 1024 * 1024); + var address = harness.MapBacked(size, ReadWrite); + var attempts = 0; + harness.Vulkan.DeviceInfo.AllocationFailure = bytes => bytes >= OptimalImageMemoryPool.DedicatedThreshold && attempts++ == 0; + try + { + var request = LinearRequest(address, size, Format.R8G8B8A8Unorm, GuestPixelFormat.Bits8_8_8_8UNorm, + GuestImageType.Color2D, new Extent3D(width, height, 1), 1, 4, 1); + var image = harness.Find(ref request); + + Assert.True(harness.Images.Contains(image)); + Assert.Equal(2, attempts); + } + finally + { + harness.Vulkan.DeviceInfo.AllocationFailure = null; + } + + harness.Shutdown(); + } + + [Fact] + public void UnsynchronizedGpuImageTest_MatchesItsRangesWithoutAllocating() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + const ulong size = 0x1000; + using var harness = new CacheHarness(_vulkan); + var address = harness.MapBacked(0x10000, ReadWrite); + var request = LinearRequest(address, size, Format.R8G8B8A8Unorm, GuestPixelFormat.Bits8_8_8_8UNorm, + GuestImageType.Color2D, new Extent3D(32, 32, 1), 1, 4, 1); + var image = harness.Acquire(ref request); + + bool Agrees(ulong probe) => harness.Worker.Run(() => + harness.Images.HasUnsynchronizedGpuImage(probe, 4) == (harness.Images.UnsynchronizedGpuImageRanges(probe, 4).Count != 0)); + Assert.True(Agrees(address + 0x40)); + Assert.False(harness.Worker.Run(() => harness.Images.HasUnsynchronizedGpuImage(address + 0x40, 4))); + + harness.Image(image).MarkGpuModified(); + Assert.True(harness.Worker.Run(() => harness.Images.HasUnsynchronizedGpuImage(address + 0x40, 4))); + Assert.True(Agrees(address + 0x40)); + Assert.True(Agrees(address + size + 0x40)); + + // Resource reads ask this for every guest word; it must not allocate. + var allocated = harness.Worker.Run(() => + { + _ = harness.Images.HasUnsynchronizedGpuImage(address + size + 0x40, 4); + var before = GC.GetAllocatedBytesForCurrentThread(); + for (var index = 0; index < 1000; index++) + { + _ = harness.Images.HasUnsynchronizedGpuImage(address + 0x40, 4); + _ = harness.Images.HasUnsynchronizedGpuImage(address + size + 0x40, 4); + } + + return GC.GetAllocatedBytesForCurrentThread() - before; + }); + Assert.Equal(0, allocated); + harness.Shutdown(); + } } diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Images/GuestPixelFormatsTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Images/GuestPixelFormatsTests.cs index 4e540e3..33c29b0 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Images/GuestPixelFormatsTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Images/GuestPixelFormatsTests.cs @@ -15,7 +15,7 @@ public sealed class GuestPixelFormatsTests Assert.Equal(4u, GuestPixelFormats.BytesPerElement(GuestPixelFormat.Bits8_8_8_8UNorm)); Assert.Equal(8u, GuestPixelFormats.BytesPerElement(GuestPixelFormat.Bits16_16_16_16Float)); Assert.Equal(2u, GuestPixelFormats.BytesPerElement(GuestPixelFormat.Bits8_8UScaled)); - Assert.Equal(0u, GuestPixelFormats.BytesPerElement(GuestPixelFormat.Bits8SNorm)); + Assert.Equal(1u, GuestPixelFormats.BytesPerElement(GuestPixelFormat.Bits8SNorm)); Assert.Equal(0u, GuestPixelFormats.BytesPerElement(GuestPixelFormat.Invalid)); Assert.Equal(0u, GuestPixelFormats.BytesPerElement((GuestPixelFormat)200)); Assert.Equal(0u, GuestPixelFormats.BytesPerElement((GuestPixelFormat)0xFFFF_FFFF)); @@ -37,7 +37,8 @@ public sealed class GuestPixelFormatsTests Assert.Equal(TextureNumericClass.Sint, GuestPixelFormats.SampledNumericClass(GuestPixelFormat.Bits8_8_8_8SInt)); Assert.Equal(TextureNumericClass.Float, GuestPixelFormats.SampledNumericClass(GuestPixelFormat.Bits8_8_8_8UNorm)); Assert.Equal(TextureNumericClass.Float, GuestPixelFormats.SampledNumericClass(GuestPixelFormat.Bits8_8UScaled)); - Assert.Equal(TextureNumericClass.Unsupported, GuestPixelFormats.SampledNumericClass(GuestPixelFormat.Bits8SNorm)); + Assert.Equal(TextureNumericClass.Float, GuestPixelFormats.SampledNumericClass(GuestPixelFormat.Bits8SNorm)); + Assert.Equal(TextureNumericClass.Sint, GuestPixelFormats.SampledNumericClass(GuestPixelFormat.Bits8SInt)); Assert.Equal(GuestPixelFormat.Bits32UInt, GuestPixelFormats.RemapTextureFormat(GuestPixelFormat.Bits11_11_10UInt)); Assert.Equal(GuestPixelFormat.Bc7Srgb, GuestPixelFormats.RemapTextureFormat(GuestPixelFormat.Bc7Srgb)); Assert.Equal(Format.R8G8B8A8Unorm, GuestPixelFormats.HostFormat(GuestPixelFormat.Bits8_8_8_8UNorm)); diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Images/ImageCacheTestSupport.cs b/tests/SharpEmu.Libs.Tests/Gpu/Images/ImageCacheTestSupport.cs index d1b29d2..2e123d2 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Images/ImageCacheTestSupport.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Images/ImageCacheTestSupport.cs @@ -148,6 +148,8 @@ internal static class ImageCacheTestSupport var extent = image.Backing.Extent; var layers = image.Backing.ImageType == ImageType.Type3D ? 1u : image.Backing.Layers; var bytesPerTexel = image.Description.BytesPerBlock; + if (image.Backing.Format is Format.R8G8B8A8Srgb or Format.R8G8B8A8Unorm) + bytesPerTexel = 4; if (aspect == ImageAspectFlags.DepthBit) { bytesPerTexel = DepthFormatRule.AspectTransferBytes(image.Backing.Format); diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Images/ImageRequestBuildersTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Images/ImageRequestBuildersTests.cs index c2cbe0a..495bbec 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Images/ImageRequestBuildersTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Images/ImageRequestBuildersTests.cs @@ -18,6 +18,53 @@ public sealed class ImageRequestBuildersTests : IClassFixture(() => ImageRequestBuilders.Texture(levelPastTheView, shape)); + } + + [Fact] + public void OnlyDescriptorsWithABaseAndAHostFormatDescribeAHostTexture() + { + Assert.True(ImageRequestBuilders.DescribesHostTexture(RegisterWords.Texture(Base, GuestPixelFormat.Bits8_8_8_8UNorm, 32, 32))); + Assert.False(ImageRequestBuilders.DescribesHostTexture(RegisterWords.Texture(Base, GuestPixelFormat.Invalid, 32, 32))); + Assert.False(ImageRequestBuilders.DescribesHostTexture(RegisterWords.Texture(0, GuestPixelFormat.Bits8_8_8_8UNorm, 32, 32))); + Assert.False(ImageRequestBuilders.DescribesHostTexture(new uint[8])); + } + [Fact] public void EightBitUnsignedScaledTextureUsesUnormBackingWithShaderConversion() { @@ -287,13 +334,34 @@ public sealed class ImageRequestBuildersTests : IClassFixture 0); Assert.NotEqual(request.Description.MipLayout[1].Offset, request.Description.MipLayout[2].Offset); - Assert.Equal(1u, request.View.BaseLevel); + Assert.Equal(0u, request.View.BaseLevel); Assert.Equal(2u, request.View.LevelCount); } + [Fact] + public void Texture_MappedMipChainIsTheWholeImageViewedFromTheBaseLevel() + { + var words = RegisterWords.Texture(Base, GuestPixelFormat.Bits8_8_8_8UNorm, 64, 64, baseLevel: 1, lastLevel: 2, maxMip: 2); + var resident = ImageRequestBuilders.Texture(words, Sampled2D).Request; + var whole = ImageRequestBuilders.WithWholeMipChain(resident); + + // The image of a rendered chain whose lower mip a pass samples: all levels, viewed from 1. + Assert.Equal(0u, whole.Description.FirstLevel); + Assert.Equal(3u, whole.Description.Resources.Levels); + Assert.Equal(resident.Description.Data, whole.Description.Data); + Assert.Equal(1u, whole.View.BaseLevel); + Assert.Equal(2u, whole.View.LevelCount); + Assert.Equal(0f, whole.View.MinLod); + var again = ImageRequestBuilders.WithWholeMipChain(whole); + Assert.Equal(whole.View, again.View); + Assert.Equal(whole.Description.Resources, again.Description.Resources); + } + [Fact] public void Texture_CubeBecomesALayeredTwoDimensionalImage() { @@ -434,6 +502,10 @@ public sealed class ImageRequestBuildersTests : IClassFixture + Assert.False(ImageRequestBuilders.TryFixedDccClearValue(code, out _)); + private readonly record struct GuestSpanCheck(ulong Address, ulong Size); } diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Images/MemoryBudgetTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Images/MemoryBudgetTests.cs new file mode 100644 index 0000000..bdc39d0 --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Gpu/Images/MemoryBudgetTests.cs @@ -0,0 +1,25 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.Libs.Gpu.Buffers; +using SharpEmu.Libs.Gpu.Images; +using Xunit; + +namespace SharpEmu.Libs.Tests.Gpu.Images; + +public sealed class MemoryBudgetTests +{ + [Fact] + public void AvailableMemoryUsesBudgetMinusCurrentUsage() + { + Assert.Equal(8UL << 30, GpuDeviceInfo.CalculateAvailableBytes(12UL << 30, 4UL << 30)); + Assert.Equal(0UL, GpuDeviceInfo.CalculateAvailableBytes(4UL << 30, 5UL << 30)); + } + + [Fact] + public void ImageCacheLeavesMarginAndCapsHeapFallback() + { + Assert.Equal(6UL << 30, GuestImageCache.ComputeImageCacheBudget(8UL << 30, hasMemoryBudget: true)); + Assert.Equal(4UL << 30, GuestImageCache.ComputeImageCacheBudget(12UL << 30, hasMemoryBudget: false)); + } +} diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Images/RenderScalePolicyTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Images/RenderScalePolicyTests.cs new file mode 100644 index 0000000..32e8344 --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Gpu/Images/RenderScalePolicyTests.cs @@ -0,0 +1,154 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.HLE.GpuMemory; +using SharpEmu.Libs.Gpu.Images; +using Silk.NET.Vulkan; +using Xunit; + +namespace SharpEmu.Libs.Tests.Gpu.Images; + +public sealed class RenderScalePolicyTests +{ + private const float Half = 0.5f; + + // Without a device nothing can be resized; these tests ask about the geometry rules. + static RenderScalePolicyTests() => RenderScalePolicy.ConfigureFormatSupport(_ => true); + + private static ImageDescription ColorTarget(uint width = 1920, uint height = 1080) + { + var description = ImageDescription.Create(); + description.Data = new GuestSpan(0x1_0000, (ulong)width * height * 4); + description.PixelFormat = Format.R8G8B8A8Unorm; + description.GuestFormat = GuestPixelFormat.Bits8_8_8_8UNorm; + description.Extent = new Extent3D(width, height, 1); + description.Pitch = width; + description.BytesPerBlock = 4; + description.TileMode = GuestTileMode.RenderTarget; + description.MipLayout[0] = new MipLevelLayout { Offset = 0, Size = description.Data.Size, Pitch = width, Height = height }; + return description; + } + + [Fact] + public void ScreenClassRenderTargetsScale() + { + Assert.Equal(Half, RenderScalePolicy.ScaleFor(ColorTarget(), Half)); + Assert.Equal(Half, RenderScalePolicy.ScaleFor(ColorTarget(3840, 2160), Half)); + Assert.Equal(2.0f, RenderScalePolicy.ScaleFor(ColorTarget(2304, 1296), 2.0f)); + } + + [Fact] + public void NativeScaleNeverChangesAnything() + { + Assert.Equal(1.0f, RenderScalePolicy.ScaleFor(ColorTarget(), 1.0f)); + Assert.Equal(1.0f, RenderScalePolicy.ScaleFor(ColorTarget(640, 360), 1.0f)); + } + + [Fact] + public void DepthTargetsScale() + { + var description = ColorTarget(); + description.PixelFormat = Format.D32Sfloat; + description.GuestFormat = GuestPixelFormat.Bits32Float; + description.TileMode = GuestTileMode.Depth; + Assert.Equal(Half, RenderScalePolicy.ScaleFor(description, Half)); + } + + [Theory] + [InlineData(1279u, 1080u)] + [InlineData(1920u, 719u)] + [InlineData(64u, 64u)] + public void SmallTargetsStayNative(uint width, uint height) + { + Assert.Equal(1.0f, RenderScalePolicy.ScaleFor(ColorTarget(width, height), Half)); + } + + [Fact] + public void SquareTargetsStayNative() + { + Assert.Equal(1.0f, RenderScalePolicy.ScaleFor(ColorTarget(2048, 2048), Half)); + } + + [Fact] + public void AssetTilingsStayNative() + { + foreach (var tileMode in new[] { GuestTileMode.Linear, GuestTileMode.Standard64KB, GuestTileMode.Standard4KB, GuestTileMode.Prt }) + { + var description = ColorTarget(); + description.TileMode = tileMode; + Assert.Equal(1.0f, RenderScalePolicy.ScaleFor(description, Half)); + } + } + + [Fact] + public void FootprintsTheHostCannotReproduceStayNative() + { + var multisample = ColorTarget(); + multisample.Samples = 4; + Assert.Equal(1.0f, RenderScalePolicy.ScaleFor(multisample, Half)); + + var mipped = ColorTarget(); + mipped.Resources = new SubresourceCount(4, 1); + Assert.Equal(1.0f, RenderScalePolicy.ScaleFor(mipped, Half)); + + var layered = ColorTarget(); + layered.Resources = new SubresourceCount(1, 6); + Assert.Equal(1.0f, RenderScalePolicy.ScaleFor(layered, Half)); + + var volume = ColorTarget(); + volume.Type = GuestImageType.Color3D; + Assert.Equal(1.0f, RenderScalePolicy.ScaleFor(volume, Half)); + + var compressed = ColorTarget(); + compressed.GuestFormat = GuestPixelFormat.Bc7UNorm; + Assert.Equal(1.0f, RenderScalePolicy.ScaleFor(compressed, Half)); + + var htile = ColorTarget(); + htile.PixelFormat = Format.D32Sfloat; + htile.GuestFormat = GuestPixelFormat.Bits32Float; + htile.TileMode = GuestTileMode.Depth; + htile.Metadata = new MetadataDescription { Kind = MetadataKind.Htile, Range = new GuestSpan(0x40_0000, 0x1000) }; + Assert.Equal(1.0f, RenderScalePolicy.ScaleFor(htile, Half)); + + var stencil = ColorTarget(); + stencil.Stencil = new GuestSpan(0x80_0000, 0x1000); + Assert.Equal(1.0f, RenderScalePolicy.ScaleFor(stencil, Half)); + } + + // DCC describes guest bytes the host image never stores, so the backing can be any size. + [Fact] + public void ColourCompressionMetadataStillScales() + { + var description = ColorTarget(); + description.Metadata = new MetadataDescription { Kind = MetadataKind.Dcc, Range = new GuestSpan(0x40_0000, 0x1000) }; + Assert.Equal(Half, RenderScalePolicy.ScaleFor(description, Half)); + } + + [Fact] + public void HelperImagesWithoutGuestMemoryStayNative() + { + var description = ColorTarget(); + description.Data = default; + Assert.Equal(1.0f, RenderScalePolicy.ScaleFor(description, Half)); + } + + [Theory] + [InlineData(1920u, 0.5f, 960u)] + [InlineData(1080u, 0.5f, 540u)] + [InlineData(1281u, 0.5f, 641u)] + [InlineData(1920u, 0.75f, 1440u)] + [InlineData(1920u, 2.0f, 3840u)] + [InlineData(1u, 0.5f, 1u)] + [InlineData(0u, 0.5f, 0u)] + public void LengthsRoundUpSoNoGuestPixelIsLost(uint length, float scale, uint expected) + { + Assert.Equal(expected, RenderScalePolicy.ScaleLength(length, scale)); + } + + [Fact] + public void ExtentScalingLeavesDepthAlone() + { + var scaled = RenderScalePolicy.ScaleExtent(new Extent3D(1920, 1080, 4), Half); + Assert.Equal(new Extent3D(960, 540, 4), scaled); + } +} diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Images/TextureTransferLayoutTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Images/TextureTransferLayoutTests.cs index 516bf59..af3eb47 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Images/TextureTransferLayoutTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Images/TextureTransferLayoutTests.cs @@ -157,4 +157,22 @@ public sealed class TextureTransferLayoutTests Assert.Throws(() => TextureTransferLayout.SurfaceFormat(GuestPixelFormat.Invalid)); Assert.Contains(fatal.Messages, message => message.Contains("has no host format")); } + + [Fact] + public void CopyFootprint_EndsAtTheLastTexelOfThePaddedRows() + { + var region = new BufferImageCopy + { + BufferOffset = 16, + BufferRowLength = 8, + ImageSubresource = new ImageSubresourceLayers(ImageAspectFlags.ColorBit, 0, 0, 1), + ImageExtent = new Extent3D(3, 2, 1), + }; + // Row 0 starts at 16, row 1 at 16 + 8 * 4; its third texel ends at 16 + 44. + Assert.Equal(60UL, TextureTransferLayout.CopyFootprint([region], new TileElementLayout(4, 1, 1))); + + var compressed = region with { BufferOffset = 0, BufferRowLength = 0, ImageExtent = new Extent3D(8, 8, 1) }; + // An 8x8 BC block image is two block rows of two 8-byte blocks. + Assert.Equal(32UL, TextureTransferLayout.CopyFootprint([compressed], new TileElementLayout(8, 4, 4))); + } } diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Images/TilerParityTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Images/TilerParityTests.cs index ef70828..32512d5 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Images/TilerParityTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Images/TilerParityTests.cs @@ -38,6 +38,63 @@ public sealed class TilerParityTests : IClassFixture return bytes; } + [Theory] + [InlineData(1u, 1u)] + [InlineData(2u, 1u)] + [InlineData(1u, 3u)] + [InlineData(2u, 3u)] + public void NarrowColorPacking_PreservesEncodedBytesAndPadding(uint channels, uint layers) + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var harness = new ImageTestHarness(_vulkan); + const uint width = 257, height = 3, sourceBase = 3, wideBase = 4, narrowBase = 1; + var narrowStride = width * channels + 5; + const uint wideStride = width * 4 + 12; + var narrowSlice = height * narrowStride + 7; + const uint wideSlice = height * wideStride + 16; + var input = new byte[sourceBase + layers * narrowSlice + 4]; + var expectedWide = Enumerable.Repeat((byte)0x5a, (int)(wideBase + layers * wideSlice + 4)).ToArray(); + var expectedNarrow = Enumerable.Repeat((byte)0x5a, (int)((narrowBase + layers * narrowSlice + 7) & ~3u)).ToArray(); + for (uint layer = 0; layer < layers; layer++) + for (uint row = 0; row < height; row++) + for (uint x = 0; x < width; x++) + { + var wide = wideBase + layer * wideSlice + row * wideStride + x * 4; + expectedWide[wide] = expectedWide[wide + 1] = expectedWide[wide + 2] = 0; + expectedWide[wide + 3] = 255; + for (uint channel = 0; channel < channels; channel++) + { + var value = (byte)(x + layer * 53 + row * 37 + channel * 91); + input[sourceBase + layer * narrowSlice + row * narrowStride + x * channels + channel] = value; + expectedWide[wide + channel] = value; + expectedNarrow[narrowBase + layer * narrowSlice + row * narrowStride + x * channels + channel] = value; + } + } + var source = harness.Upload(input); + var wideOutput = harness.CreateDeviceLocalBuffer((ulong)expectedWide.Length); + var layout = new DepthConversionLayout { Width = width, Height = height, Layers = layers, SourceRowStride = narrowStride, TargetRowStride = wideStride, + SourceSliceStride = narrowSlice, TargetSliceStride = wideSlice }; + harness.Run(() => + { + wideOutput.Fill(0, wideOutput.Size, 0x5a5a5a5a); + harness.Tiler.ConvertNarrowColor(new TilerBufferSpan(source.Handle, sourceBase, source.Size - sourceBase), + new TilerBufferSpan(wideOutput.Handle, wideBase, wideOutput.Size - wideBase), channels, true, layout); + }); + Assert.Equal(expectedWide, harness.ReadBack(wideOutput.Handle, 0, wideOutput.Size)); + var wideSource = harness.Upload(expectedWide); + var narrowOutput = harness.CreateDeviceLocalBuffer((ulong)expectedNarrow.Length); + layout = new DepthConversionLayout { Width = width, Height = height, Layers = layers, SourceRowStride = wideStride, TargetRowStride = narrowStride, + SourceSliceStride = wideSlice, TargetSliceStride = narrowSlice }; + harness.Run(() => + { + narrowOutput.Fill(0, narrowOutput.Size, 0x5a5a5a5a); + harness.Tiler.ConvertNarrowColor(new TilerBufferSpan(wideSource.Handle, wideBase, wideSource.Size - wideBase), + new TilerBufferSpan(narrowOutput.Handle, narrowBase, narrowOutput.Size - narrowBase), channels, false, layout); + }); + Assert.Equal(expectedNarrow, harness.ReadBack(narrowOutput.Handle, 0, narrowOutput.Size)); + harness.AssertNoValidationMessages(); + } + [Fact] public void BlockCopies_ProduceTheSameBytesAsTheReferenceModules() { diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/BindlessImageHeapCapacityTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/BindlessImageHeapCapacityTests.cs new file mode 100644 index 0000000..deb5a40 --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/BindlessImageHeapCapacityTests.cs @@ -0,0 +1,55 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.Libs.Gpu.Pipelines; +using SharpEmu.Libs.Tests.Gpu.Scheduling; +using Xunit; + +namespace SharpEmu.Libs.Tests.Gpu.Pipelines; + +[Collection(SchedulingStateCollection.Name)] +public sealed class BindlessImageHeapCapacityTests +{ + private static BindlessImageHeap.Limits Limits(uint images = uint.MaxValue, uint samplers = uint.MaxValue, uint total = uint.MaxValue) => + new(images, images, samplers, images, images, samplers, images, images, samplers, total); + + [Fact] + public void SamplerLimits_BoundTheSamplerBinding() + { + // A device with few samplers per stage (MoltenVK without argument buffers reports 16). + var (_, _, samplers) = BindlessImageHeap.Capacities(Limits(samplers: 16)); + + Assert.Equal(16u, samplers); + } + + [Fact] + public void Samplers_CountAgainstThePoolWideTotal() + { + var (sampled, storage, samplers) = BindlessImageHeap.Capacities(Limits(total: 5096)); + + Assert.Equal(4096u, samplers); + Assert.Equal(5096u, sampled + storage + samplers); + Assert.Equal(500u, sampled); + Assert.Equal(500u, storage); + } + + [Fact] + public void UnboundedDevices_KeepTheStableCapacities() + { + // NVIDIA reports the pool-wide total as UINT_MAX. + var (sampled, storage, samplers) = BindlessImageHeap.Capacities(Limits()); + + Assert.Equal(4096u, samplers); + Assert.Equal(128u * 1024, sampled); + Assert.Equal(128u * 1024, storage); + } + + [Fact] + public void NoSamplerOrImageRoom_IsAnExplicitFailure() + { + using var fatal = new FatalScope(); + + Assert.Throws(() => BindlessImageHeap.Capacities(Limits(samplers: 0))); + Assert.Throws(() => BindlessImageHeap.Capacities(Limits(total: 4096))); + } +} diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/BoundedFillDetectorTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/BoundedFillDetectorTests.cs index c48839a..d8e1eb6 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/BoundedFillDetectorTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/BoundedFillDetectorTests.cs @@ -169,6 +169,58 @@ public sealed class BoundedFillDetectorTests Assert.Null(BoundedFillDetector.DetectCopy(Decode(ZeroRangeFromConstantBuffer))); } + [Fact] + public void APatternWithUnverifiedArithmeticIsRejected() + { + var program = Decode(PatternFromUserData); + program = program with + { + Instructions = program.Instructions.Select(instruction => instruction.Opcode == "VRcpIflagF32" + ? instruction with { Sources = [instruction.Sources[0] with { Kind = Gen5OperandKind.EncodedConstant, Value = 128 }] } + : instruction).ToArray(), + }; + Assert.Null(BoundedFillDetector.Detect(program)); + } + + [Theory] + [InlineData(0)] // reciprocal scale + [InlineData(1)] // overwrite the output index + [InlineData(2)] // restore another EXEC mask + [InlineData(3)] // float negation + public void APatternWithChangedDataFlowIsRejected(int mutation) + { + var program = Decode(PatternFromUserData); + var instructions = program.Instructions.ToArray(); + var at = Array.FindIndex(instructions, instruction => instruction.Opcode == + (mutation == 0 ? "VMulF32" : mutation == 1 ? "VSubI32" : mutation == 2 ? "SMovB64" : "VLshlAddU32")); + var original = instructions[at]; + instructions[at] = mutation switch + { + 0 => original with { Sources = [original.Sources[0] with { Value = 0x3F800000 }, original.Sources[1]] }, + 1 => original with { Destinations = [Gen5Operand.Vector(2)] }, + 2 => original with { Sources = [Gen5Operand.Scalar(8)] }, + _ => original with { Control = ((Gen5Vop3Control)original.Control!) with { NegateMask = 1 } }, + }; + Assert.Null(BoundedFillDetector.Detect(program with { Instructions = instructions })); + } + + [Fact] + public void ThePatternIsRecognizedAfterVectorRegisterAllocationChanges() + { + var program = Decode(PatternFromUserData); + // Preserve the architectural input v0 and move only the global index v2. + Gen5Operand Rename(Gen5Operand operand) => operand.Kind == Gen5OperandKind.VectorRegister && operand.Value == 2 + ? operand with { Value = 17 } : operand; + var renamed = program.Instructions.Select(instruction => instruction with + { + Sources = instruction.Sources.Select(Rename).ToArray(), + Destinations = instruction.Destinations.Select(Rename).ToArray(), + Control = instruction.Control is Gen5BufferMemoryControl control + ? control with { VectorAddress = 17 } : instruction.Control, + }).ToArray(); + Assert.NotNull(BoundedFillDetector.Detect(program with { Instructions = renamed })); + } + [Fact] public void APatternFillWithAnotherBranchTargetIsRejected() { diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/ComputeRejectionPolicyTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/ComputeRejectionPolicyTests.cs index d8d1b28..776160b 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/ComputeRejectionPolicyTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/ComputeRejectionPolicyTests.cs @@ -81,6 +81,9 @@ public sealed class ComputeRejectionPolicyTests var failure = Assert.Throws(() => guest.Programs.TryGetProgram( source, PipelineTestGuest.ComputeOptions(1), false, ref cursor, out _, out _, out _)); Assert.Contains("could not be materialized", failure.Message); + // The unreadable pointer is user data [0, 2], i.e. guest address 0x200000000. + Assert.Contains("detail=", failure.Message); + Assert.Contains("0x0000000200000000", failure.Message); Assert.Empty(guest.Compiler.Requests); } diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/ConstantFillDetectorTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/ConstantFillDetectorTests.cs new file mode 100644 index 0000000..71c9ac3 --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/ConstantFillDetectorTests.cs @@ -0,0 +1,40 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.Libs.Gpu.Pipelines; +using SharpEmu.ShaderCompiler; +using Xunit; + +namespace SharpEmu.Libs.Tests.Gpu.Pipelines; + +public sealed class ConstantFillDetectorTests +{ + private static Gen5ShaderProgram ImmediateProgram() => new(0, + [ + new(0, default, "VLshlAddU32", [], [Gen5Operand.Scalar(8), + new(Gen5OperandKind.EncodedConstant, 134), Gen5Operand.Vector(0)], [Gen5Operand.Vector(2)], null), + new(8, default, "VMovB32", [], [new(Gen5OperandKind.LiteralConstant, 0x40404040)], [Gen5Operand.Vector(1)], null), + new(12, default, "BufferStoreFormatX", [], [Gen5Operand.Vector(2), Gen5Operand.Scalar(0), + new(Gen5OperandKind.EncodedConstant, 128)], [Gen5Operand.Vector(1)], + new Gen5BufferMemoryControl(1, 2, 1, 0, 0, true, false, false, false)), + new(20, default, "SEndpgm", [], [], [], null), + ]); + + [Fact] + public void AnImmediateStoreAtGroupTimes64PlusThreadIsRecognized() + { + Assert.Equal(new ImmediateConstantFill(8, 0), ConstantFillDetector.DetectImmediate(ImmediateProgram())); + } + + [Fact] + public void AConstantStoreAtAnotherAddressIsRejected() + { + var program = ImmediateProgram(); + var instructions = program.Instructions.ToArray(); + instructions[2] = instructions[2] with + { + Control = ((Gen5BufferMemoryControl)instructions[2].Control!) with { VectorAddress = 0 }, + }; + Assert.Null(ConstantFillDetector.DetectImmediate(program with { Instructions = instructions })); + } +} diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/DescriptorHeapTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/DescriptorHeapTests.cs index 9e1171e..a405717 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/DescriptorHeapTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/DescriptorHeapTests.cs @@ -88,6 +88,55 @@ public sealed unsafe class DescriptorHeapTests(HeadlessVulkanFixture fixture) : vulkan.AssertNoValidationMessages(); } + // Ordinary pools stay small; a set too large for one gets a pool sized for a few of its kind. + [Fact] + public void PoolCapacity_GrowsOnlyForSetsThatDoNotFit() + { + Assert.Equal(DescriptorHeap.PoolCapacity, DescriptorHeap.CapacityFor(new DescriptorSetDemand(SampledImages: 64))); + var large = DescriptorHeap.CapacityFor(new DescriptorSetDemand(SampledImages: 65536, Samplers: 2)); + Assert.Equal(4u * 65536, large.SampledImages); + Assert.Equal(DescriptorHeap.PoolCapacity.StorageBuffers, large.StorageBuffers); + Assert.Equal(DescriptorHeap.PoolCapacity.Samplers, large.Samplers); + } + + [Fact] + public void SetLargerThanAnOrdinaryPool_GetsALargerPool() + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan)) + { + return; + } + + const uint samplers = 2000; + Assert.True(samplers > DescriptorHeap.PoolCapacity.Samplers); + using var fatal = new FatalScope(); + using var harness = new ImageTestHarness(vulkan); + var layout = CreateSamplerLayout(harness, samplers); + DescriptorHeap? heap = null; + try + { + harness.Run(() => + { + heap = new DescriptorHeap(harness.Device, harness.Scheduler); + Assert.Equal(DescriptorHeap.PoolCapacity, heap.CurrentCapacity); + Assert.Equal(3, Commit(heap, layout, 3, samplers).Count); + Assert.True(heap.CurrentCapacity.Samplers >= samplers); + }); + } + finally + { + harness.Finish(); + harness.Run(() => + { + heap?.Dispose(); + harness.Vk.DestroyDescriptorSetLayout(harness.Device.Device, layout, null); + }); + } + + vulkan.AssertNoValidationMessages(); + } + [Fact] public void RetiredPool_IsResetOnlyAfterItsTickCompletes() { diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/GraphicsRejectionPolicyTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/GraphicsRejectionPolicyTests.cs index f6f3fc2..f8080f9 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/GraphicsRejectionPolicyTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/GraphicsRejectionPolicyTests.cs @@ -74,8 +74,8 @@ public sealed class GraphicsRejectionPolicyTests 0xF0000108, 0x00010400, 0xBF810000]); for (var index = 0; index < 33; index++) guest.WriteWords(TableAddress + 312 + (ulong)index * 32, - distinctCandidates ? 0x1000u + (uint)index : 0x1000u, 20u << 20, 0, - 0xFACu | ((!distinctCandidates && index == 1 ? 10u : 9u) << 28), 0, 0, 0, 0); + distinctCandidates ? 0x1000u + (uint)index : 0x1000u, (!distinctCandidates && index == 1 ? GuestImageFormat.Format11x2x10Uint : 20u) << 20, 0, + 0xFACu | (9u << 28), 0, 0, 0, 0); var source = guest.Source(PixelAddress, ShaderStage.Pixel, [(uint)(TableAddress & uint.MaxValue), (uint)(TableAddress >> 32)]); var options = new StageCompileOptions { PixelInfo = new PixelInputInfo() }; var cursor = 7u; diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/PipelineCacheStreamTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/PipelineCacheStreamTests.cs new file mode 100644 index 0000000..90b2d0c --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/PipelineCacheStreamTests.cs @@ -0,0 +1,73 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Runtime.InteropServices; +using System.Text; +using SharpEmu.Libs.Gpu.Pipelines; +using Xunit; + +namespace SharpEmu.Libs.Tests.Gpu.Pipelines; + +// Large titles' driver caches pass 2 GiB, so they are written and read as streams from +// native memory. The file format stays the one Wrap and TryUnwrap use. +public sealed unsafe class PipelineCacheStreamTests +{ + private static readonly string Signature = + PipelineCacheSignature.Build(0x10DE, 0x2C05, 0x0123, new byte[PipelineCacheSignature.UuidSize]); + + private static byte[] Payload() => Encoding.ASCII.GetBytes("driver cache bytes, streamed"); + + [Fact] + public void StreamedFile_RoundTrips() + { + var payload = Payload(); + using var stream = new MemoryStream(); + fixed (byte* data = payload) + PipelineCacheSignature.WriteTo(stream, Signature, data, (ulong)payload.Length); + + stream.Position = 0; + Assert.True(PipelineCacheSignature.TryReadFrom(stream, Signature, out var read, out var length)); + try + { + Assert.Equal(payload, new ReadOnlySpan(read, (int)length).ToArray()); + } + finally + { + NativeMemory.Free(read); + } + } + + // Caches saved by earlier builds keep loading. + [Fact] + public void FileWrittenWithWrap_ReadsAsAStream() + { + var payload = Payload(); + using var stream = new MemoryStream(PipelineCacheSignature.Wrap(Signature, payload)); + Assert.True(PipelineCacheSignature.TryReadFrom(stream, Signature, out var read, out var length)); + try + { + Assert.Equal(payload, new ReadOnlySpan(read, (int)length).ToArray()); + } + finally + { + NativeMemory.Free(read); + } + } + + [Fact] + public void CorruptedPayload_IsRejected() + { + var file = PipelineCacheSignature.Wrap(Signature, Payload()); + file[^1] ^= 0xFF; + using var stream = new MemoryStream(file); + Assert.False(PipelineCacheSignature.TryReadFrom(stream, Signature, out _, out _)); + } + + [Fact] + public void OtherDevice_IsRejected() + { + var other = PipelineCacheSignature.Build(0x1002, 0x73BF, 0x0123, new byte[PipelineCacheSignature.UuidSize]); + using var stream = new MemoryStream(PipelineCacheSignature.Wrap(other, Payload())); + Assert.False(PipelineCacheSignature.TryReadFrom(stream, Signature, out _, out _)); + } +} diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/PipelineTestSupport.cs b/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/PipelineTestSupport.cs index f3742b7..d568c58 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/PipelineTestSupport.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/PipelineTestSupport.cs @@ -45,6 +45,12 @@ internal sealed class FakePipelineHost(ICpuMemory memory) : IShaderPipelineHost public bool GraphicsSubgroupOperationsEnabled => true; public bool SharedInt64AtomicsEnabled => false; + public bool PostDepthCoverageSupported { get; set; } + + public bool ExactFloat16ConversionsEnabled { get; set; } + public bool NonUniformImageIndexingEnabled { get; set; } + public bool RuntimeBufferStridesEnabled { get; set; } + public bool UsesBindlessImages { get; set; } public ShaderPrewarmList? ShaderPrewarm { get; set; } diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/ShaderPipelineCacheTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/ShaderPipelineCacheTests.cs index 4d2a98b..9a99bac 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/ShaderPipelineCacheTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/ShaderPipelineCacheTests.cs @@ -19,6 +19,24 @@ public sealed class ShaderPipelineCacheTests : IDisposable { private const uint Format32x4Float = 77; + [Theory] + [InlineData(6u)] + [InlineData(32u)] + public void MergedHullUserData_CarriesTheBackAddressBesideTheDeclaredFrontBank(uint count) + { + var registers = new UserScalarRegisters(); + for (uint index = 0; index < count; index++) registers.Set(index, 100 + index, UserScalarKind.Unknown); + var data = ShaderPipelineCache.MergedHullUserData(registers, count, 0x20_ABCDEFBC); + Assert.Equal((int)count + 2, data.Length); + Assert.Equal(registers.Values.AsSpan(0, (int)count).ToArray(), data.AsSpan(0, (int)count).ToArray()); + Assert.Equal(0xABCDEFBCu, data[count]); + Assert.Equal(0x20u, data[count + 1]); + var changed = ShaderPipelineCache.MergedHullUserData(registers, count, 0x21_98765404); + Assert.Equal(0x98765404u, changed[count]); + Assert.Equal(0x21u, changed[count + 1]); + Assert.Equal(0xABCDEFBCu, data[count]); + } + private readonly FatalScope _fatal = new(); public void Dispose() => _fatal.Dispose(); @@ -132,6 +150,47 @@ public sealed class ShaderPipelineCacheTests : IDisposable Assert.False(description.StaticParameters.CullBack); } + [Theory] + [InlineData(true, 15, 0, 1, 0, true)] + [InlineData(true, 1, 16, 1, 0, true)] + [InlineData(true, 1, 0, 17, 0, true)] + [InlineData(true, 1, 0, 1, 18, true)] + [InlineData(true, 4, 5, 1, 0, false)] + [InlineData(false, 15, 16, 17, 18, false)] + public void SecondBlendSource_IsUsedOnlyByEnabledSrc1Factors(bool enable, byte colorSource, byte colorDestination, + byte alphaSource, byte alphaDestination, bool expected) + { + var blend = new BlendRegisters + { + Enable = enable, ColorSourceFactor = colorSource, ColorDestinationFactor = colorDestination, + AlphaSourceFactor = alphaSource, AlphaDestinationFactor = alphaDestination, + }; + + Assert.Equal(expected, ShaderPipelineCache.UsesSecondBlendSource(blend)); + } + + [Fact] + public void TessellatedPatches_KeepTheGuestCulling() + { + var banks = Banks(); + banks.Context.RasterMode.CullBack = true; + var programs = Programs(); + var tessellated = new VertexInputInfo + { + PositionExportControl = programs.VertexInput.PositionExportControl, + Stage = programs.VertexInput.Stage, + Tessellation = new(new(10), new(20), 4), + }; + var rendering = new RenderingState { Samples = 1 }; + + var description = ShaderPipelineCache.BuildGraphicsDescription( + [], default, tessellated, programs.PixelInput, banks.Context, in rendering, PrimitiveTopology.PatchList, false, false, + programs.Vertex, programs.Pixel, SampleCountFlags.Count1Bit); + + Assert.Equal(PrimitiveTopology.PatchList, description.StaticParameters.Topology); + Assert.True(description.StaticParameters.CullBack); + } + [Fact] public void DepthTarget_FoldsTheDepthAndStencilState() { @@ -169,6 +228,17 @@ public sealed class ShaderPipelineCacheTests : IDisposable Assert.Equal(new PipelineVertexAttribute(0, 1), state.Attributes[2]); } + // Depth bounds are dynamic state: a game that moves them per draw (light volumes) must not + // create a pipeline for every value. + [Fact] + public void DepthBounds_DoNotSplitTheGraphicsPipeline() + { + var near = Describe(banks => { banks.Context.DepthBoundsMin = 0.1f; banks.Context.DepthBoundsMax = 0.4f; }, withDepth: true); + var far = Describe(banks => { banks.Context.DepthBoundsMin = 0.6f; banks.Context.DepthBoundsMax = 0.9f; }, withDepth: true); + + Assert.Equal(ShaderPipelineCache.KeyOf(near), ShaderPipelineCache.KeyOf(far)); + } + [Fact] public void GraphicsPipelineKey_FoldsAll166StaticBytes() { @@ -212,6 +282,25 @@ public sealed class ShaderPipelineCacheTests : IDisposable Assert.Equal(key, ShaderPipelineCache.KeyOf(With(description, description.StaticParameters))); } + [Fact] + public void GraphicsPipelineKey_DistinguishesTessellationStagesAndPatchSizes() + { + var basic = Describe(); + GraphicsPipelineKey Key(ulong control, ulong evaluation, uint points) => ShaderPipelineCache.KeyOf(new() + { + Rendering = basic.Rendering, VertexInput = basic.VertexInput, VertexInfo = basic.VertexInfo, + VertexProgram = basic.VertexProgram, VertexStage = basic.VertexStage, PixelInfo = basic.PixelInfo, + PixelProgram = basic.PixelProgram, PixelStage = basic.PixelStage, StaticParameters = basic.StaticParameters, + Tessellation = new(new(control), new(evaluation), points), + }); + var key = Key(10, 20, 4); + Assert.Equal(key, Key(10, 20, 4)); + Assert.NotEqual(key, ShaderPipelineCache.KeyOf(basic)); + Assert.NotEqual(key, Key(11, 20, 4)); + Assert.NotEqual(key, Key(10, 21, 4)); + Assert.NotEqual(key, Key(10, 20, 3)); + } + [Fact] public void ComputePipelineKey_IsTheProgramId() { @@ -231,6 +320,28 @@ public sealed class ShaderPipelineCacheTests : IDisposable Assert.Equal(2, cache.ComputePipelineCount); } + [Fact] + public void RenderingKeysDistinguishColorAndDepthSampleCounts() + { + PipelineRenderingState State(uint color, uint depth) + { + var state = new PipelineRenderingState { ColorCount = 1, DepthFormat = Format.D32Sfloat, DepthSamples = depth }; + state.ColorFormats[0] = Format.R8G8B8A8Unorm; + state.ColorSamples[0] = color; + return state; + } + var entries = new Dictionary + { + [State(1, 1)] = 1, + [State(1, 2)] = 2, + [State(2, 2)] = 3, + }; + Assert.Equal(3, entries.Count); + Assert.Equal(1, entries[State(1, 1)]); + Assert.Equal(2, entries[State(1, 2)]); + Assert.Equal(3, entries[State(2, 2)]); + } + [Fact] public void TryCreateComputePipeline_ReportsTheHostCompileAndCachesTheResult() { @@ -315,7 +426,7 @@ public sealed class ShaderPipelineCacheTests : IDisposable var programs = Programs(pixelStage: Stage(new ShaderProgramInfo { Stage = ShaderStageKind.Pixel, PixelColorExportMasks = exportMasks })); var resolution = new ColorTargetResolution( default, 0x1000, 0x10000, new Extent2D(64, 64), 0, 0, 1, ColorComponentMap.Identity, false, false, default); - ColorTargetState[] colors = [new(in resolution, 0, new SharpEmu.Libs.Gpu.Buffers.ResourceSlotIdentifier(1, 1))]; + ColorTargetState[] colors = [new(in resolution, 0, new SharpEmu.Libs.Gpu.Buffers.ResourceSlotIdentifier(1, 1), 1f)]; var rendering = new RenderingState { Samples = 1, ColorAttachmentCount = 1 }; rendering.ColorAttachments[0] = new RenderingAttachment( default, ImageLayout.ColorAttachmentOptimal, Format.R8G8B8A8Unorm, 0, 0, 0, 0, false, false, false, false, false); @@ -335,7 +446,7 @@ public sealed class ShaderPipelineCacheTests : IDisposable var programs = Programs(); var resolution = new ColorTargetResolution( default, 0x1000, 0x10000, new Extent2D(64, 64), 0, 0, 1, ColorComponentMap.Identity, false, false, default); - ColorTargetState[] colors = [new(in resolution, 0, new SharpEmu.Libs.Gpu.Buffers.ResourceSlotIdentifier(1, 1))]; + ColorTargetState[] colors = [new(in resolution, 0, new SharpEmu.Libs.Gpu.Buffers.ResourceSlotIdentifier(1, 1), 1f)]; var rendering = new RenderingState { Samples = 1, ColorAttachmentCount = 1 }; rendering.ColorAttachments[0] = new RenderingAttachment( default, ImageLayout.ColorAttachmentOptimal, Format.R8G8B8A8Unorm, 0, 0, 0, 0, false, false, false, false, false); @@ -381,6 +492,30 @@ public sealed class ShaderPipelineCacheTests : IDisposable Assert.Equal(expected, description.StaticParameters.SampleShadingEnable); } + [Theory] + [InlineData(0xFFFFFFFFu, 0xFFFFFFFFu, true)] + [InlineData(0x00030003u, 0x00030003u, true)] + [InlineData(0xFFFFFFFEu, 0xFFFFFFFFu, false)] + [InlineData(0xFFFEFFFFu, 0xFFFFFFFFu, false)] + [InlineData(0xFFFFFFFFu, 0xFFFFFFFEu, false)] + [InlineData(0xFFFFFFFFu, 0xFFFEFFFFu, false)] + public void NativeTwoSampleRenderingRejectsRestrictedCoverage(uint firstRow, uint secondRow, bool supported) + { + using var fatalScope = new FatalScope(); + var banks = Banks(); + banks.Context.AntialiasingConfig.SampleCountLog2 = 1; + banks.Context.SampleCoverageMaskX0Y0X1Y0 = firstRow; + banks.Context.SampleCoverageMaskX0Y1X1Y1 = secondRow; + var programs = Programs(); + var rendering = new RenderingState { Samples = 2 }; + GraphicsPipelineDescription Describe() => ShaderPipelineCache.BuildGraphicsDescription( + [], default, programs.VertexInput, programs.PixelInput, banks.Context, in rendering, + PrimitiveTopology.TriangleList, false, false, programs.Vertex, programs.Pixel, + SampleCountFlags.Count2Bit, nativeTwoSampleMixedSupported: true); + if (supported) Assert.Equal(2u, Describe().StaticParameters.Samples); + else Assert.Contains("restricted per-pixel coverage", Assert.Throws(() => Describe()).Message); + } + [Fact] public void MissingPrograms_AreFatal() { diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/ShaderPlanningDumpTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/ShaderPlanningDumpTests.cs index 02ae012..f3fae14 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/ShaderPlanningDumpTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/ShaderPlanningDumpTests.cs @@ -30,9 +30,11 @@ public sealed class ShaderPlanningDumpTests Environment.SetEnvironmentVariable(names[2], outputDirectory); using var fatalScope = new FatalScope(); var guest = new PipelineTestGuest(); - // A lane-derived offset reads an image descriptor without a material-table selector. + // A lane-derived offset reads the descriptor of an image store without a material-table + // selector; only plan-time descriptors can bind a store. Adding one to its first word + // keeps an empty source buffer from proving a null descriptor. guest.RegisterProgram(codeAddress, PipelineTestGuest.MemoryBase + 0x8000, - [0x7E100500, 0xF42C0402, 0x10000000, 0xF0000108, 0x00040000, 0xBF810000]); + [0x7E100500, 0xF42C0402, 0x10000000, 0x80108110, 0xF0200108, 0x00040000, 0xBF810000]); var source = guest.Source(codeAddress, ShaderStage.Compute, new uint[8]); var cursor = 0u; var failure = Assert.Throws(() => diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/ShaderPrewarmListTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/ShaderPrewarmListTests.cs index 7629dce..dd78c74 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/ShaderPrewarmListTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/ShaderPrewarmListTests.cs @@ -60,8 +60,8 @@ public sealed class ShaderPrewarmListTests : IDisposable var (record, code) = Assert.Single(reloaded.LoadedComputes()); Assert.True( ShaderProgramCache.TryCompilePrewarm( - record, code, new FakeShaderCompiler(Compile), host.SharedInt64AtomicsEnabled, host.ExecGuardElisionEnabled, - host.ShaderSignedZeroInfNanPreserveFloat32Supported, out var compiled, out var layout, out var error), + record, code, new FakeShaderCompiler(Compile), host, + out var compiled, out var layout, out var error), error); Assert.NotNull(layout); Assert.Equal(runtime, Assert.IsType(compiled).Spirv); @@ -83,6 +83,38 @@ public sealed class ShaderPrewarmListTests : IDisposable Assert.Same(loaded[0].Code.Ranges, loaded[1].Code.Ranges); } + [Fact] + public void PrewarmUsesTheRuntimeDeviceFeaturesAndBindingLayout() + { + var guest = new PipelineTestGuest(Compile); + guest.Host.ExactFloat16ConversionsEnabled = true; + guest.Host.NonUniformImageIndexingEnabled = true; + guest.Host.RuntimeBufferStridesEnabled = true; + guest.Host.UsesBindlessImages = true; + using (var list = Open()) + { + guest.Host.ShaderPrewarm = list; + guest.RegisterProgram(CodeAddress, HeaderAddress, PipelineTestGuest.FormatLoadProgram); + var cursor = 0u; + guest.Programs.GetOrCompile(guest.Source(CodeAddress, ShaderStage.Compute, + PipelineTestGuest.BufferDescriptor(BufferAddress, 4, 64, BufferDescriptorWords.Format32UInt)), + PipelineTestGuest.ComputeOptions(threadsX: 64), ref cursor, out _); + } + + using var reloaded = Open(); + var (record, code) = Assert.Single(reloaded.LoadedComputes()); + var compiler = new FakeShaderCompiler(Compile); + Assert.True(ShaderProgramCache.TryCompilePrewarm(record, code, compiler, + guest.Host, + out var compiled, out var layout, out var error), error); + var request = Assert.Single(compiler.Shaders).Request; + Assert.True(request.SupportsExactFloat16Conversions); + Assert.True(request.SupportsNonUniformImageIndexing); + Assert.True(layout!.UsesBindlessImages); + Assert.True(layout.UsesRuntimeBufferStrides); + Assert.Equal(Assert.Single(guest.Compiler.Shaders).Spirv, Assert.IsType(compiled).Spirv); + } + [Fact] public void ATruncatedLastRecordIsDroppedAndTheListKeepsAppending() { @@ -109,6 +141,26 @@ public sealed class ShaderPrewarmListTests : IDisposable Assert.Equal(2, reloaded.LoadedComputeCount); } + [Fact] + public void APrewarmFileFromAnOlderPlannerVersionIsDiscarded() + { + using (var list = Open()) + { + CompileAtRuntime(list, BufferDescriptorWords.Format32UInt); + } + + var path = Path.Combine(_directory, ShaderPrewarmList.FileName); + using (var stream = new FileStream(path, FileMode.Open, FileAccess.Write, FileShare.Read)) + { + stream.Position = sizeof(uint); + stream.Write(BitConverter.GetBytes(1u)); + } + + using var reloaded = Open(); + Assert.Empty(reloaded.LoadedComputes()); + Assert.Equal(3 * sizeof(uint), new FileInfo(path).Length); + } + [Fact] public void TheStampMatchesOnlyTheTextLastWritten() { diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/ShaderProgramCacheTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/ShaderProgramCacheTests.cs index 8ac17c8..76c2be2 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/ShaderProgramCacheTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/ShaderProgramCacheTests.cs @@ -4,6 +4,7 @@ using SharpEmu.Libs.Gpu.Pipelines; using SharpEmu.Libs.Gpu.Rendering; using SharpEmu.Libs.Tests.Gpu.Scheduling; +using SharpEmu.ShaderCompiler; using SharpEmu.ShaderCompiler.Resources; using Xunit; @@ -13,6 +14,146 @@ namespace SharpEmu.Libs.Tests.Gpu.Pipelines; [Collection(SchedulingStateCollection.Name)] public sealed class ShaderProgramCacheTests : IDisposable { + [Theory] + [InlineData(1u, 4)] + [InlineData(2u, 8)] + [InlineData(0x10u, 4)] + [InlineData(0x20u, 8)] + public void CustomSampleOffsetsAreUsedOnlyForSampleInterpolation(uint activeInput, int expectedCount) + { + _guest.RegisterProgram(CodeA, HeaderA, [0xBF810000]); + var source = _guest.Source(CodeA, ShaderStage.Pixel, []); + var shaderInterface = new SharpEmu.Libs.Gpu.GpuCommands.Registers.ShaderInterfaceRegisters + { + PixelInputEnable = activeInput, + PixelInputAddress = activeInput, + }; + var offsets = Enumerable.Repeat((X: -.25f, Y: 0f), expectedCount).ToArray(); + var info = PixelStageInputResolver.Resolve(_guest.Context, source.Registered, shaderInterface, + new byte[8], new SharpEmu.Libs.Gpu.Images.ColorComponentMap[8], 0, + rasterizationSamples: 2, customSampleOffsets: offsets); + Assert.Equal((activeInput & 0x11u) != 0 ? expectedCount : 0, info.CustomSampleOffsets.Count); + } + + [Fact] + public void CustomSamplePositionsReachCompilerAndSeparateCachedPrograms() + { + _guest.RegisterProgram(CodeA, HeaderA, [0xBF810000]); + var source = _guest.Source(CodeA, ShaderStage.Pixel, []); + ShaderProgram Compile(float x) + { + var cursor = 0u; + return _guest.Programs.GetOrCompile(source, + new StageCompileOptions { PixelInfo = new PixelInputInfo + { InterpolationSample = 0, CustomSampleOffsets = Enumerable.Repeat((X: x, Y: 0f), 4).ToArray() } }, + ref cursor, out _); + } + var left = Compile(-.25f); + var right = Compile(.25f); + Assert.NotEqual(left, right); + Assert.Equal(left, Compile(-.25f)); + Assert.Equal(-.25f, _guest.Compiler.Requests[0].PixelCustomSampleOffsets[0].X); + Assert.Equal(.25f, _guest.Compiler.Requests[1].PixelCustomSampleOffsets[0].X); + } + [Theory] + [InlineData(0u, 0u)] + [InlineData(1u, 0xFFFEu)] + [InlineData(2u, 0u)] + [InlineData(3u, 0u)] + public void SampleExclusionIsInactiveOutsideEarlyDepthOrder(uint order, uint expectedMask) + { + _guest.RegisterProgram(CodeA, HeaderA, [0xBF810000]); + var source = _guest.Source(CodeA, ShaderStage.Pixel, []); + var shaderInterface = new SharpEmu.Libs.Gpu.GpuCommands.Registers.ShaderInterfaceRegisters + { + DepthShaderControl = SharpEmu.Libs.Gpu.GpuCommands.Registers.DepthShaderControlRegisters.Decode(order << 4), + }; + var info = PixelStageInputResolver.Resolve(_guest.Context, source.Registered, shaderInterface, + new byte[8], new SharpEmu.Libs.Gpu.Images.ColorComponentMap[8], 0, + rasterizationSamples: 2, pixelShaderIterationSamples: 1, shaderSampleExclusionMask: 0xABCDFFFE); + Assert.Equal(expectedMask, info.ShaderSampleExclusionMask); + } + + [Theory] + [InlineData(0x110u, true)] + [InlineData(0x50u, false)] + public void ShaderCoverageChangesUseLateDepthWithoutForcedDepthBeforeShader(uint raw, bool maskExport) + { + _guest.RegisterProgram(CodeA, HeaderA, [0xBF810000]); + var source = _guest.Source(CodeA, ShaderStage.Pixel, []); + var shaderInterface = new SharpEmu.Libs.Gpu.GpuCommands.Registers.ShaderInterfaceRegisters + { + DepthShaderControl = SharpEmu.Libs.Gpu.GpuCommands.Registers.DepthShaderControlRegisters.Decode(raw), + }; + var info = PixelStageInputResolver.Resolve(_guest.Context, source.Registered, shaderInterface, + new byte[8], new SharpEmu.Libs.Gpu.Images.ColorComponentMap[8], 0, + maskExportSamples: 2, rasterizationSamples: 2, pixelShaderIterationSamples: 2, + shaderSampleExclusionMask: 0xFFFE); + Assert.False(info.EarlyDepth); + Assert.Equal(maskExport, info.SampleMaskExportEnable); + Assert.Equal(2u, info.MaskExportSamples); + Assert.Equal(0u, info.ShaderSampleExclusionMask); + } + [Theory] + [InlineData(0x11u, false, true)] + [InlineData(0x51u, false, true)] + [InlineData(0x111u, false, true)] + [InlineData(0x151u, false, true)] + [InlineData(0x51u, true, false)] + [InlineData(0x50u, true, false)] + [InlineData(0x1011u, false, false)] + [InlineData(0x211u, false, false)] + [InlineData(0x411u, false, false)] + [InlineData(0x8011u, false, false)] + public void DepthExportsUseLateFallbackOnlyWhenGuestOrderingIsNotForced(uint raw, bool forced, bool accepted) + { + _guest.RegisterProgram(CodeA, HeaderA, [0xBF810000]); + var source = _guest.Source(CodeA, ShaderStage.Pixel, []); + var shaderInterface = new SharpEmu.Libs.Gpu.GpuCommands.Registers.ShaderInterfaceRegisters { + DepthShaderControl = SharpEmu.Libs.Gpu.GpuCommands.Registers.DepthShaderControlRegisters.Decode(raw) }; + PixelInputInfo Resolve() => PixelStageInputResolver.Resolve(_guest.Context, source.Registered, shaderInterface, + new byte[8], new SharpEmu.Libs.Gpu.Images.ColorComponentMap[8], 0, + rasterizationSamples: 2, pixelShaderIterationSamples: 1, shaderSampleExclusionMask: 0xFFFE, + forceShaderDepthOrder: forced); + if (!accepted) + { + Assert.Throws(() => Resolve()); + return; + } + var info = Resolve(); + Assert.False(info.EarlyDepth); + Assert.True(info.DepthExportEnable); + Assert.Equal((raw & 0x40) != 0, info.KillEnable); + Assert.Equal((raw & 0x100) != 0, info.SampleMaskExportEnable); + Assert.Equal(0u, info.ShaderSampleExclusionMask); + var cursor = 0u; + _guest.Programs.GetOrCompile(source, new StageCompileOptions { PixelInfo = info }, ref cursor, out _); + var request = Assert.Single(_guest.Compiler.Requests); + Assert.True(request.PixelDepthExportEnable); + Assert.False(request.EarlyFragmentTests); + Assert.Equal(info.SampleMaskExportEnable, request.PixelSampleMaskExportEnable); + } + + [Fact] + public void SampleExclusionRequiresSupportAndSeparatesEarlyPixelPrograms() + { + _guest.RegisterProgram(CodeA, HeaderA, [0xBF810000]); + var source = _guest.Source(CodeA, ShaderStage.Pixel, []); + ShaderProgram Compile(uint mask) + { + var cursor = 0u; + return _guest.Programs.GetOrCompile(source, + new StageCompileOptions { PixelInfo = new PixelInputInfo + { EarlyDepth = true, ShaderSampleExclusionMask = mask } }, ref cursor, out _); + } + Assert.Throws(() => Compile(0xFFFE)); + _guest.Host.PostDepthCoverageSupported = true; + var enabled = Compile(0xFFFE); + Assert.NotEqual(enabled, Compile(0)); + Assert.Equal(enabled, Compile(0xFFFE)); + Assert.Contains(_guest.Compiler.Requests, request => request.PixelShaderSampleExclusionMask == 0xFFFE); + } + private const ulong CodeA = PipelineTestGuest.MemoryBase + 0x1000; private const ulong CodeB = PipelineTestGuest.MemoryBase + 0x2000; private const ulong HeaderA = PipelineTestGuest.MemoryBase + 0x8000; @@ -21,6 +162,149 @@ public sealed class ShaderProgramCacheTests : IDisposable private const uint Format32x4Uint = 75; private const uint Format32x4Float = 77; + [Fact] + public void TessellationRuntimeData_UsesOnePermutationAcrossDifferentPushCursors() + { + _guest.RegisterProgram(CodeA, HeaderA, PipelineTestGuest.EndProgram); + var source = _guest.Source(CodeA, ShaderStage.TessellationEvaluation, []); + var options = new StageCompileOptions + { + VertexInfo = new VertexInputInfo(), + Tessellation = new(Gen5TessellationDomain.Quads, Gen5TessellationSpacing.Equal, false, false), + }; + var cursor = 0u; + var first = _guest.Programs.GetOrCompile(source, options, ref cursor, out var firstStage); + cursor = 9; + var second = _guest.Programs.GetOrCompile(source, options, ref cursor, out var secondStage); + Assert.Equal(first, second); + Assert.Same(firstStage.Program, secondStage.Program); + Assert.Single(_guest.Compiler.Requests); + Assert.Single(Assert.Single(_guest.Programs.Entries).Permutations); + Assert.True(firstStage.Program!.Bindings!.UsesTessellationData); + Assert.False(firstStage.Program.Bindings.UsesPushData); + Assert.Equal(9u, cursor); + } + + [Fact] + public void EarlyDepthStateReachesCompilerAndSeparatesCachedPrograms() + { + _guest.RegisterProgram(CodeA, HeaderA, [0xBF810000]); + var source = _guest.Source(CodeA, ShaderStage.Pixel, []); + ShaderProgram CompilePixel(bool early) + { + var cursor = 0u; + return _guest.Programs.GetOrCompile(source, + new StageCompileOptions { PixelInfo = new PixelInputInfo { EarlyDepth = early } }, + ref cursor, out _); + } + var late = CompilePixel(false); + var early = CompilePixel(true); + Assert.NotEqual(late, early); + Assert.Equal(late, CompilePixel(false)); + Assert.Equal(new[] { false, true }, _guest.Compiler.Requests.Select(request => request.EarlyFragmentTests)); + } + + [Fact] + public void DepthExportEnableReachesCompilerAndSeparatesCachedPrograms() + { + _guest.RegisterProgram(CodeA, HeaderA, [0xBF810000]); + var source = _guest.Source(CodeA, ShaderStage.Pixel, []); + ShaderProgram CompilePixel(bool depthExport) + { + var cursor = 0u; + return _guest.Programs.GetOrCompile(source, + new StageCompileOptions { PixelInfo = new PixelInputInfo { DepthExportEnable = depthExport } }, + ref cursor, out _); + } + var disabled = CompilePixel(false); + var enabled = CompilePixel(true); + Assert.NotEqual(disabled, enabled); + Assert.Equal(enabled, CompilePixel(true)); + Assert.Equal(new[] { false, true }, _guest.Compiler.Requests.Select(request => request.PixelDepthExportEnable)); + } + + [Theory] + [InlineData(1u, 1u, 2u, false, 0u)] + [InlineData(16u, 1u, 2u, false, 0u)] + [InlineData(1u, 2u, 2u, true, uint.MaxValue)] + [InlineData(2u, 2u, 2u, true, uint.MaxValue)] + [InlineData(2u, 1u, 2u, false, uint.MaxValue)] + public void PixelIterationCountIsIndependentOfSampleQualifiedInputs(uint inputs, uint iterations, + uint raster, bool sampleShading, uint fixedSample) + { + _guest.RegisterProgram(CodeA, HeaderA, [0xBF810000]); + var source = _guest.Source(CodeA, ShaderStage.Pixel, []); + var shaderInterface = new SharpEmu.Libs.Gpu.GpuCommands.Registers.ShaderInterfaceRegisters + { PixelInputEnable = inputs, PixelInputAddress = inputs }; + var info = PixelStageInputResolver.Resolve(_guest.Context, source.Registered, shaderInterface, + new byte[8], new SharpEmu.Libs.Gpu.Images.ColorComponentMap[8], 0, + rasterizationSamples: raster, pixelShaderIterationSamples: iterations); + Assert.Equal(sampleShading, info.SampleShading); + Assert.Equal(fixedSample == uint.MaxValue ? null : (uint?)fixedSample, info.InterpolationSample); + var cursor = 0u; + _guest.Programs.GetOrCompile(source, new StageCompileOptions { PixelInfo = info }, ref cursor, out _); + Assert.Equal(info.InterpolationSample, Assert.Single(_guest.Compiler.Requests).PixelInterpolationSample); + } + + [Theory] + [InlineData(0u, 2u)] + [InlineData(4u, 2u)] + [InlineData(2u, 4u)] + public void UnverifiedPixelIterationMappingsAreRejected(uint iterations, uint raster) + { + _guest.RegisterProgram(CodeA, HeaderA, [0xBF810000]); + var source = _guest.Source(CodeA, ShaderStage.Pixel, []); + var failure = Assert.Throws(() => PixelStageInputResolver.Resolve( + _guest.Context, source.Registered, new(), new byte[8], + new SharpEmu.Libs.Gpu.Images.ColorComponentMap[8], 0, + rasterizationSamples: raster, pixelShaderIterationSamples: iterations)); + Assert.Contains("pixel-shader iteration counts", failure.Message); + } + + [Fact] + public void ExplicitInterpolationSampleSeparatesCachedPixelPrograms() + { + _guest.RegisterProgram(CodeA, HeaderA, [0xBF810000]); + var source = _guest.Source(CodeA, ShaderStage.Pixel, []); + ShaderProgram Compile(uint? sample) + { + var cursor = 0u; + return _guest.Programs.GetOrCompile(source, + new StageCompileOptions { PixelInfo = new PixelInputInfo { InterpolationSample = sample } }, ref cursor, out _); + } + var pixel = Compile(null); var fixedSample = Compile(0); + Assert.NotEqual(pixel, fixedSample); + Assert.Equal(fixedSample, Compile(0)); + Assert.Equal(new uint?[] { null, 0 }, _guest.Compiler.Requests.Select(request => request.PixelInterpolationSample)); + } + + [Fact] + public void SampleMaskExportCountsReachCompilerAndSeparateCachedPrograms() + { + _guest.RegisterProgram(CodeA, HeaderA, [0xBF810000]); + var source = _guest.Source(CodeA, ShaderStage.Pixel, []); + ShaderProgram CompilePixel(uint exportSamples, uint rasterSamples) + { + var cursor = 0u; + return _guest.Programs.GetOrCompile(source, + new StageCompileOptions { PixelInfo = new PixelInputInfo + { + SampleMaskExportEnable = true, + MaskExportSamples = exportSamples, + RasterizationSamples = rasterSamples, + } }, ref cursor, out _); + } + var broadcast = CompilePixel(1, 2); + var independent = CompilePixel(2, 2); + var single = CompilePixel(1, 1); + Assert.NotEqual(broadcast, independent); + Assert.NotEqual(broadcast, single); + Assert.Equal(independent, CompilePixel(2, 2)); + Assert.All(_guest.Compiler.Requests, request => Assert.True(request.PixelSampleMaskExportEnable)); + Assert.Equal(new[] { (1u, 2u), (2u, 2u), (1u, 1u) }, _guest.Compiler.Requests + .Select(request => (request.PixelMaskExportSamples, request.PixelRasterizationSamples))); + } + [Fact] public void EmbeddedFetchIsReplacedBeforeTheProgramRequestsResourceTables() { @@ -149,6 +433,31 @@ public sealed class ShaderProgramCacheTests : IDisposable Assert.All(_guest.Programs.Entries, entry => Assert.Single(entry.Permutations)); } + [Fact] + public void PixelExportFormatChange_CompilesANewProgramForTheSameTargetKind() + { + _guest.RegisterProgram(CodeA, HeaderA, PipelineTestGuest.EndProgram); + var source = _guest.Source(CodeA, ShaderStage.Pixel, []); + ShaderProgram CompileFormat(Gen5PixelExportFormat format) + { + var formats = new byte[PixelInputInfo.TargetCount]; + formats[0] = (byte)format; + var options = new StageCompileOptions + { + PixelInfo = new PixelInputInfo { TargetExportFormats = formats }, + PixelOutputs = [new Gen5PixelOutputBinding(0, 0, Gen5PixelOutputKind.Float) { ExportFormat = format }], + }; + var cursor = 0u; + return _guest.Programs.GetOrCompile(source, options, ref cursor, out _); + } + var half = CompileFormat(Gen5PixelExportFormat.Float16); + var normalized = CompileFormat(Gen5PixelExportFormat.Unorm16); + Assert.NotEqual(half.Id, normalized.Id); + Assert.Equal(normalized.Id, CompileFormat(Gen5PixelExportFormat.Unorm16).Id); + Assert.Equal(2, _guest.Compiler.Requests.Count); + Assert.Equal(Gen5PixelExportFormat.Unorm16, _guest.Compiler.Requests[1].PixelOutputs[0].ExportFormat); + } + [Fact] public void SpecializationChange_IsANewPermutationOfTheSameEntry() { diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/VertexInputResolverTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/VertexInputResolverTests.cs index 63b750e..de20b9d 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/VertexInputResolverTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Pipelines/VertexInputResolverTests.cs @@ -354,4 +354,36 @@ public sealed class VertexInputResolverTests : IDisposable Assert.Contains("records=4/5", fatal.Message); } + [Theory] + [InlineData(74u, 3, 12ul)] + [InlineData(77u, 4, 16ul)] + [InlineData(77u, 3, 16ul)] + [InlineData(64u, 2, 8ul)] + public void MergeBuffers_ConstantUncheckedStreamIncludesFullFormattedFetch(uint format, int components, ulong expectedSize) + { + var resource = Attribute(StreamBase, 0, 1, 0); + resource = resource with + { + Descriptor = resource.Descriptor with { Word3 = IdentitySelect | (format << 12) | (2u << 28) }, + RegisterCount = components, + }; + var buffer = Assert.Single(VertexInputResolver.MergeBuffers([resource], ShaderAddress)); + Assert.Equal(1u, buffer.RecordCount); + Assert.Equal(0u, buffer.Stride); + Assert.Equal(expectedSize, buffer.Size); + } + + [Theory] + [InlineData(0u, 1u)] + [InlineData(1u, 1u)] + [InlineData(3u, 1u)] + [InlineData(2u, 0u)] + public void MergeBuffers_BoundedOrUnboundConstantStreamKeepsDeclaredFootprint(uint outOfBounds, uint records) + { + var resource = Attribute(StreamBase, 0, records, 0); + resource = resource with { Descriptor = resource.Descriptor with { Word3 = resource.Descriptor.Word3 | (outOfBounds << 28) } }; + var buffer = Assert.Single(VertexInputResolver.MergeBuffers([resource], ShaderAddress)); + Assert.Equal((ulong)records, buffer.Size); + } + } diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Rendering/RenderExecutorComputeTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Rendering/RenderExecutorComputeTests.cs index b05928c..ce5d790 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Rendering/RenderExecutorComputeTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Rendering/RenderExecutorComputeTests.cs @@ -1,6 +1,7 @@ // Copyright (C) 2026 SharpEmu Emulator Project // SPDX-License-Identifier: GPL-2.0-or-later +using System.Buffers.Binary; using SharpEmu.Libs.Gpu.Rendering; using SharpEmu.Libs.Tests.Gpu.Scheduling; using Xunit; @@ -106,6 +107,34 @@ public sealed class RenderExecutorComputeTests : IDisposable AssertNotDispatched(); Assert.DoesNotContain("end_rendering", _host.Calls); Assert.DoesNotContain("reset_bindings", _host.Calls); + Assert.Empty(_pipelines.Calls); + } + + [Fact] + public void IndirectDispatch_WithCleanZeroArguments_SkipsProgramLookup() + { + var address = RecordingRenderHost.MemoryBase + 0x400; + Span arguments = stackalloc byte[3 * sizeof(uint)]; + BinaryPrimitives.WriteUInt32LittleEndian(arguments[sizeof(uint)..], 1); + BinaryPrimitives.WriteUInt32LittleEndian(arguments[(2 * sizeof(uint))..], 1); + _host.WriteGuest(address, arguments); + + _executor.Dispatch(1, Banks(), 1, 1, 1, 0x41, address); + + Assert.Empty(_pipelines.Calls); + Assert.DoesNotContain("end_rendering", _host.Calls); + } + + [Fact] + public void IndirectDispatch_WithGpuOwnedArguments_StillCompilesAndDispatches() + { + var address = RecordingRenderHost.MemoryBase + 0x400; + _host.WriteGuest(address, new byte[3 * sizeof(uint)]); + _host.CleanGuestMemoryAvailable = false; + + _executor.Dispatch(1, Banks(), 1, 1, 1, 0x41, address); + + AssertDispatched(1, 1, 1); } [Fact] @@ -195,18 +224,22 @@ public sealed class RenderExecutorComputeTests : IDisposable } // AGC's fill kernel: s_buffer_load_dword the value from a one-record buffer, store it per thread. - private void ConfigureLoadedValueFill(uint records, ulong valueAddress, uint value) + private void ConfigureLoadedValueFill(uint records, ulong valueAddress, uint value, bool recognized = true) { Span bytes = stackalloc byte[sizeof(uint)]; System.Buffers.Binary.BinaryPrimitives.WriteUInt32LittleEndian(bytes, value); Assert.True(_host.GuestMemory.TryWrite(valueAddress, bytes)); var target = BufferDescriptor(MetadataAddress, 4, records, format: BufferDescriptorWords.Format32UInt); var source = BufferDescriptor(valueAddress, 0, 1); - var program = Program(ShaderStageKind.Compute, buffers: - [ + var program = new ShaderProgramInfo + { + Stage = ShaderStageKind.Compute, + ConstantFill = recognized ? new SharpEmu.Libs.Gpu.Pipelines.ConstantFill(0, 0, 4) : null, + Buffers = [ new BufferResourceInfo(Read: true, Written: false, Atomic: false, Formatted: false, Scalar: true, MaxByteExtent: 4, PackedStride: 0), ClearResource(maxByteExtent: 4, packedStride: 4), - ]); + ], + }; _pipelines.Compute = ComputeProgram(Stage(program, buffers: [source, target], userData: [.. target, .. source])); } @@ -225,6 +258,27 @@ public sealed class RenderExecutorComputeTests : IDisposable Assert.Null(_executor.TryDecodeImageClear(_pipelines.Compute.Input, 128, 2, 1, 0x41)); } + [Fact] + public void BufferUsageAlone_DoesNotProveAConstantFill() + { + ConfigureLoadedValueFill(128 * 64, RecordingRenderHost.MemoryBase + 0x70_0000, 0x4040_4040, recognized: false); + Assert.Null(_executor.TryDecodeImageClear(_pipelines.Compute.Input, 128, 1, 1, 0x41)); + } + + [Fact] + public void ConstantStoreValueAlone_DoesNotProveEveryRecordIsWritten() + { + var descriptor = BufferDescriptor(MetadataAddress, 4, 128 * 64, format: BufferDescriptorWords.Format32UInt); + var program = new ShaderProgramInfo + { + Stage = ShaderStageKind.Compute, + ConstantStoreValue = 0, + Buffers = [ClearResource(maxByteExtent: 4, packedStride: 4)], + }; + var input = ComputeProgram(Stage(program, buffers: [descriptor])).Input; + Assert.Null(_executor.TryDecodeImageClear(input, 128, 1, 1, 0x41)); + } + [Fact] public void DifferentPatternDwords_RefuseTheClearAndRunTheDispatch() { diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Rendering/RenderExecutorDrawTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Rendering/RenderExecutorDrawTests.cs index aa772f4..f3d1104 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Rendering/RenderExecutorDrawTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Rendering/RenderExecutorDrawTests.cs @@ -50,11 +50,11 @@ public sealed class RenderExecutorDrawTests : IDisposable "bind_target 1", "prepare_bindings Vertex -> 1", "prepare_bindings Pixel -> 2", + "acquire_color 0 100000000 image=1", "bind_resources 1", "bind_resources 2", "obtain 100400000 40 written=False -> 100:0", "obtain 100500000 C written=False", - "acquire_color 0 100000000 image=1", "debug DrawIndex 7 100 6 0 2 0", "bind_vertex 100:0", "commit Graphics A1 [1,2]", @@ -112,6 +112,44 @@ public sealed class RenderExecutorDrawTests : IDisposable Assert.DoesNotContain(_host.Calls, c => c.StartsWith("draw_indexed_indirect", StringComparison.Ordinal)); } + [Fact] + public void IndirectAutoDraw_LetsTheGpuReadTheArguments() + { + var arguments = Auto(1, source: DrawOffsetSource.IndirectArguments) with { IndirectArgumentsAddress = IndirectArguments }; + _executor.DrawAuto(7, Banks(), arguments); + + AssertOrder("obtain 100600000 10 written=False", "begin_rendering", "bind_pipeline Graphics A1", "draw_indirect", "reset_bindings"); + Assert.DoesNotContain(_host.Calls, c => c.StartsWith("draw ", StringComparison.Ordinal)); + } + + [Fact] + public void IndirectAutoTriangleFan_ReadsTheArgumentsOnTheCpu() + { + WriteIndirectArguments(3, 2, 4, 1); + var arguments = Auto(1, source: DrawOffsetSource.IndirectArguments) with { IndirectArgumentsAddress = IndirectArguments }; + _executor.DrawAuto(7, Banks(PrimitiveTriangleFan), arguments); + + Assert.DoesNotContain(_host.Calls, c => c.StartsWith("draw_indirect", StringComparison.Ordinal)); + Assert.Contains(_host.Calls, c => c.StartsWith("draw ", StringComparison.Ordinal)); + } + + // The merged tessellation path counts patches from the vertex count on the CPU, so an + // indirect tessellation draw must read its arguments there, not leave them to the GPU. + [Fact] + public void IndirectAutoTessellation_ReadsTheArgumentsOnTheCpu() + { + WriteIndirectArguments(6, 2, 0, 0); + var banks = Banks(); + banks.Context.ShaderStages = TessellationStages; + banks.Context.ShaderInterface.TessellationFactorParameter = 0x45; + banks.Context.ShaderInterface.LocalHullConfiguration = 0xC301; + var arguments = Auto(1, source: DrawOffsetSource.IndirectArguments) with { IndirectArgumentsAddress = IndirectArguments }; + _executor.DrawAuto(7, banks, arguments); + + Assert.Contains("debug DrawIndexAuto 7 6 0 0 2 0", _host.Calls); + Assert.DoesNotContain(_host.Calls, c => c.StartsWith("draw_indirect", StringComparison.Ordinal)); + } + [Fact] public void AutoDraw_RecordsThePhasesAndTheVertexOffsets() { @@ -193,6 +231,27 @@ public sealed class RenderExecutorDrawTests : IDisposable { "max output per subgroup", banks => banks.Context.ShaderInterface.MaxOutputPerSubgroup = 0x41 }, }; + // LS/HS/ES with passthrough primitive generation: the merged tessellation path. + private const uint TessellationStages = 0x0200210D; + + [Theory] + [InlineData(0x45u, 0xC301u, "power-of-two partitioning")] + [InlineData(0x03u, 0xC301u, "an undefined domain")] + [InlineData(0x41u, 0x0000u, "an empty hull workgroup")] + public void UnsupportedTessellation_SkipsTheDrawInsteadOfFailing(uint factorParameter, uint hullConfiguration, string reason) + { + var banks = Banks(); + banks.Context.ShaderStages = TessellationStages; + banks.Context.ShaderInterface.TessellationFactorParameter = factorParameter; + banks.Context.ShaderInterface.LocalHullConfiguration = hullConfiguration; + + _executor.DrawIndexed(1, banks, Indexed(3)); + _executor.DrawAuto(2, banks, Auto(3)); + + Assert.True(_pipelines.Calls.Count == 0, reason); + Assert.DoesNotContain(_host.Calls, c => c.StartsWith("find_image", StringComparison.Ordinal)); + } + [Theory] [MemberData(nameof(UnsupportedGeometryStages))] public void UnsupportedGeometryStage_SkipsTheDraw(string reason, Action mutate) @@ -259,6 +318,9 @@ public sealed class RenderExecutorDrawTests : IDisposable Assert.False(_executor.ResolveTopology(new UserConfigRegisters { PrimitiveType = 0 }, autoDraw: false, out _)); Assert.True(_executor.ResolveTopology(new UserConfigRegisters { PrimitiveType = 17 }, autoDraw: true, out var topology)); Assert.Equal(PrimitiveTopology.TriangleStrip, topology); + // Several legacy rectangles take the rectangle-list path. + Assert.True(_executor.ResolveTopology(new UserConfigRegisters { PrimitiveType = 17 }, autoDraw: true, out topology, vertexCount: 63)); + Assert.Equal(PrimitiveTopology.PatchList, topology); var fatal = Assert.Throws(() => _executor.ResolveTopology(new UserConfigRegisters { PrimitiveType = 17 }, autoDraw: false, out _)); Assert.Contains("primitiveType=17", fatal.Message); fatal = Assert.Throws(() => _executor.ResolveTopology(new UserConfigRegisters { PrimitiveType = 9 }, autoDraw: true, out _)); @@ -508,6 +570,21 @@ public sealed class RenderExecutorDrawTests : IDisposable Assert.Contains("bind_vertex 100:100,100:0,101:0,1:0,100:180", _host.Calls); } + [Theory] + [InlineData(0x100UL)] + [InlineData(0x1200UL)] + public void IndexAllocationDoesNotRetireTheVertexBufferBeforeTheDraw(ulong vertexOffset) + { + _pipelines.Graphics = Programs(vertexBuffers: + [new VertexInputBuffer(IndexBase + vertexOffset, 16, 16)]); + _executor.DrawIndexed(1, Banks(), Indexed(4096)); + + var binding = Assert.Single(_host.LastVertexBindings); + Assert.True(_host.IsBufferLive(binding.Handle), + $"The draw binds a retired vertex buffer: {string.Join(" | ", _host.Calls)}"); + Assert.Equal(vertexOffset, binding.Offset); + } + [Fact] public void VertexRanges_ClampToTheMappedSizeAndFailForAnAddressOutsideTheAcquiredRange() { @@ -528,13 +605,16 @@ public sealed class RenderExecutorDrawTests : IDisposable } [Fact] - public void LegacyRectangleList_DrawsFourVerticesAndRejectsOtherShapes() + public void LegacyRectangleList_DrawsFourVerticesOrABatchAndRejectsOtherShapes() { _executor.DrawAuto(1, Banks(primitiveType: 17), Auto(3, instances: 2)); Assert.Contains("draw 4 2 0 0", _host.Calls); - var fatal = Assert.Throws(() => _executor.DrawAuto(1, Banks(primitiveType: 17), Auto(6))); - Assert.Contains("count=6 buffers=0", fatal.Message); + _executor.DrawAuto(1, Banks(primitiveType: 17), Auto(63)); + Assert.Contains("draw 63 1 0 0", _host.Calls); + + var fatal = Assert.Throws(() => _executor.DrawAuto(1, Banks(primitiveType: 17), Auto(5))); + Assert.Contains("count=5 buffers=0", fatal.Message); fatal = Assert.Throws(() => _executor.DrawIndexed(1, Banks(primitiveType: 17), Indexed(3))); Assert.Contains("primitiveType=17", fatal.Message); @@ -584,6 +664,23 @@ public sealed class RenderExecutorDrawTests : IDisposable Assert.DoesNotContain(_host.Calls, c => c.StartsWith("begin_rendering", StringComparison.Ordinal)); } + [Fact] + public void TargetlessPixelStorageWrite_IsExecutedEvenWhenColorExportsAreInactive() + { + var banks = Banks(); + banks.Context.RenderTargetMask = 0; + banks.Context.ShaderInterface.ColorShaderMask = 0; + var storage = new ImageResourceInfo(ImageResourceClass.Storage, true); + _pipelines.Graphics = Programs(pixelStage: Stage(Program(ShaderStageKind.Pixel, images: [storage]))); + + _executor.DrawIndexed(1, banks, Indexed(3)); + + Assert.Contains("begin_rendering 16384x8192x1 colors=0 samples=1", _host.Calls); + Assert.Contains(_host.Calls, c => c.StartsWith("prepare_bindings Pixel", StringComparison.Ordinal)); + Assert.True(Assert.Single(_pipelines.PipelineRequests).PixelActive); + Assert.Contains(_host.Calls, c => c.StartsWith("draw_indexed", StringComparison.Ordinal)); + } + [Fact] public void PixelShaderWithDepthSideEffects_KeepsADrawWithoutColorOutput() { diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Rendering/RenderExecutorStateTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Rendering/RenderExecutorStateTests.cs index a046fdc..f604455 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Rendering/RenderExecutorStateTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Rendering/RenderExecutorStateTests.cs @@ -205,6 +205,21 @@ public sealed class RenderExecutorStateTests : IDisposable Assert.Equal(new ScissorRectangle(0, 0, 0, 0), RenderExecutor.ResolveScissor(viewport, in scanMode, 64, 32)); } + [Fact] + public void ScaledScissorRoundsOutwardAndStaysInsideTheAttachment() + { + // A guest rectangle keeps its host coverage: the edges round outward, so no guest + // pixel the game asked for is clipped away by the mapping itself. + Assert.Equal(new ScissorRectangle(5, 2, 51, 29), RenderExecutor.ScaleScissor(new ScissorRectangle(11, 5, 101, 57), 0.5f, 64, 36)); + Assert.Equal(new ScissorRectangle(22, 10, 202, 114), RenderExecutor.ScaleScissor(new ScissorRectangle(11, 5, 101, 57), 2f, 256, 144)); + + // Rounding outward must not push the rectangle past the attachment. + Assert.Equal(new ScissorRectangle(0, 0, 32, 18), RenderExecutor.ScaleScissor(new ScissorRectangle(0, 0, 65, 37), 0.5f, 32, 18)); + + // An empty scissor stays empty rather than growing a row. + Assert.Equal(new ScissorRectangle(5, 5, 5, 5), RenderExecutor.ScaleScissor(new ScissorRectangle(10, 10, 10, 10), 0.5f, 64, 36)); + } + [Fact] public void Scissor_ClipRectangleRulesIntersectTheSelectedRectangles() { @@ -322,6 +337,25 @@ public sealed class RenderExecutorStateTests : IDisposable Assert.Contains("create_graphics_pipeline colors=0 depth=True topology=TriangleList restart=False", _pipelines.Calls); } + [Theory] + [InlineData(false)] + [InlineData(true)] + public void Attachments_DisabledColorBufferIgnoresStaleTargetsAndPreservesDepthEffects(bool exportsDepth) + { + var banks = Banks(withDepth: true); + banks.Context.ColorControl = ColorControlRegisters.Decode(0); + banks.Context.ColorTargets[0] = RegisterWords.Color(ColorBase, 32, 32); + banks.Context.ShaderInterface.DepthShaderControl = new DepthShaderControlRegisters { DepthExportEnable = exportsDepth }; + + _executor.DrawIndexed(1, banks, Indexed(3)); + + var rendering = Assert.Single(_host.BegunRenderings); + Assert.Equal((0u, 64u, 64u), (rendering.ColorAttachmentCount, rendering.Width, rendering.Height)); + Assert.Equal(Format.D32Sfloat, rendering.DepthFormat); + Assert.Equal(exportsDepth, Assert.Single(_pipelines.PipelineRequests).PixelActive); + Assert.True(Assert.Single(_host.DynamicStates).DepthWriteEnabled); + } + [Theory] [InlineData(0x0u, 0u, 64u)] [InlineData(0xFu, 1u, 32u)] @@ -383,6 +417,29 @@ public sealed class RenderExecutorStateTests : IDisposable Assert.Contains("imageSamples=2 targetSamples=1", fatal.Message); } + [Theory] + [InlineData(false, 1, false)] + [InlineData(true, 1, true)] + [InlineData(true, 2, false)] + public void Attachments_TwoSampleDepthWithSingleSampleColorRequiresExactHostSupport(bool supported, byte rasterLog2, bool accepted) + { + var banks = Banks(withDepth: true); + banks.Context.DepthTarget = RegisterWords.Depth(DepthBase, 64, 64, samplesLog2: 1); + banks.Context.AntialiasingConfig.SampleCountLog2 = rasterLog2; + _host.NativeTwoSampleMixedSupported = supported; + if (!accepted) + { + var fatal = Assert.Throws(() => _executor.DrawIndexed(1, banks, Indexed(3))); + Assert.Contains("color=1 depth=2", fatal.Message); + Assert.Empty(_host.BegunRenderings); + return; + } + _executor.DrawIndexed(1, banks, Indexed(3)); + var rendering = Assert.Single(_host.BegunRenderings); + Assert.Equal(2u, rendering.Samples); + Assert.Equal(1u, rendering.ColorAttachmentCount); + Assert.True(rendering.DepthStencilAttachment.HasDepth); + } [Fact] public void Attachments_SingleSampleColorWithMultisampledDepthIsFatal() { @@ -588,10 +645,10 @@ public sealed class RenderExecutorStateTests : IDisposable AssertOrder( "preparation_begin", "prepare_bindings Vertex", - "bind_resources 2", - "upload_transient 6 align=16", "wrap_refused acquire_color", "acquire_color 0", + "bind_resources 2", + "upload_transient 6 align=16", "commit Graphics", "draw_indexed 3 1 0 0 0", "preparation_end", diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Rendering/RenderExecutorTestSupport.cs b/tests/SharpEmu.Libs.Tests/Gpu/Rendering/RenderExecutorTestSupport.cs index c50241a..3a01cbd 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Rendering/RenderExecutorTestSupport.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Rendering/RenderExecutorTestSupport.cs @@ -7,6 +7,7 @@ using SharpEmu.Libs.Gpu.GpuCommands.Registers; using SharpEmu.Libs.Gpu.Images; using SharpEmu.Libs.Gpu.Rendering; using SharpEmu.Libs.Gpu.Scheduling; +using SharpEmu.HLE.GpuMemory; using SharpEmu.Libs.Tests.Gpu.Images; using SharpEmu.Libs.Tests.Gpu.Scheduling; using Xunit; @@ -93,6 +94,7 @@ internal sealed class RecordingRenderHost : IRenderHost public IImageFormatSupport FormatSupport { get; } = new AcceptingFormatSupport(); public bool IsRecording => Recording; + public bool NativeTwoSampleMixedSupported { get; set; } public ImageLayout ColorLayout { get; set; } = ImageLayout.ColorAttachmentOptimal; @@ -158,6 +160,11 @@ internal sealed class RecordingRenderHost : IRenderHost return GuestMemory.TryRead(address, destination); } + public bool CleanGuestMemoryAvailable { get; set; } = true; + + public bool TryReadCleanGuestBytes(ulong address, Span destination) => + CleanGuestMemoryAvailable && GuestMemory.TryRead(address, destination); + public ulong ClampMappedSize(ulong address, ulong size) => ClampOverride?.Invoke(address, size) ?? size; public ResourceSlotIdentifier FindImage(ref ImageRequest request, bool exactFormat) @@ -177,6 +184,10 @@ internal sealed class RecordingRenderHost : IRenderHost public void BindRenderTarget(ResourceSlotIdentifier image) => Calls.Add($"bind_target {image.Index}"); + public void DemoteRenderScale(ResourceSlotIdentifier image) => Calls.Add($"demote_scale {image.Index}"); + + public float GetRenderScale(ResourceSlotIdentifier image) => 1f; + public void ResetBindings() => Calls.Add("reset_bindings"); public ColorAttachmentAcquisition AcquireColorAttachment(in ColorTargetState target) @@ -222,6 +233,35 @@ internal sealed class RecordingRenderHost : IRenderHost public BufferBinding ObtainBuffer(ulong address, ulong size, bool isWritten) { TryWrapRing("obtain"); + var owner = ResolveAllocation(address, size); + var end = address + size; + for (var page = address & ~(PageSize - 1); page < end; page += PageSize) + { + if (_dirtyPages.Remove(page)) + { + Calls.Add($"upload {owner.Handle:X} {page:X}"); + } + } + + Calls.Add($"obtain {address:X} {size:X} written={isWritten} -> {owner.Handle:X}:{address - owner.Start:X}"); + return new BufferBinding(owner.Handle, address - owner.Start); + } + + public void PrepareBufferAllocations(ReadOnlySpan ranges) + { + for (var index = 0; index < ranges.Length; index++) + { + var range = ranges[index]; + var overlaps = _allocations.Any(allocation => !allocation.Deleted && + allocation.Start < range.End && range.Address < allocation.End); + for (var other = 0; !overlaps && other < ranges.Length; other++) + overlaps = other != index && ranges[other].Address < range.End && range.Address < ranges[other].End; + if (overlaps) _ = ResolveAllocation(range.Address, range.Size); + } + } + + private Allocation ResolveAllocation(ulong address, ulong size) + { var end = address + size; var owner = _allocations.Find(a => !a.Deleted && a.Start <= address && end <= a.End); if (owner is null) @@ -244,16 +284,7 @@ internal sealed class RecordingRenderHost : IRenderHost _allocations.Add(owner); } - for (var page = address & ~(PageSize - 1); page < end; page += PageSize) - { - if (_dirtyPages.Remove(page)) - { - Calls.Add($"upload {owner.Handle:X} {page:X}"); - } - } - - Calls.Add($"obtain {address:X} {size:X} written={isWritten} -> {owner.Handle:X}:{address - owner.Start:X}"); - return new BufferBinding(owner.Handle, address - owner.Start); + return owner; } public BufferBinding UploadTransient(ReadOnlySpan data, uint alignment) @@ -267,8 +298,15 @@ internal sealed class RecordingRenderHost : IRenderHost public byte[]? LastTransient { get; private set; } - public void BindVertexBuffers(ReadOnlySpan bindings, VertexInputInfo input) => + public BufferBinding[] LastVertexBindings { get; private set; } = []; + + public bool IsBufferLive(ulong handle) => _allocations.Any(allocation => allocation.Handle == handle && !allocation.Deleted); + + public void BindVertexBuffers(ReadOnlySpan bindings, VertexInputInfo input) + { + LastVertexBindings = bindings.ToArray(); Calls.Add($"bind_vertex {string.Join(",", bindings.ToArray().Select(b => $"{b.Handle:X}:{b.Offset:X}"))}"); + } public void BindIndexBuffer(BufferBinding binding, IndexType type) => Calls.Add($"bind_index {binding.Handle:X}:{binding.Offset:X} {type}"); @@ -325,6 +363,9 @@ internal sealed class RecordingRenderHost : IRenderHost public void DrawIndexedIndirect(BufferBinding arguments) => Calls.Add($"draw_indexed_indirect {arguments.Handle:X}:{arguments.Offset:X}"); + public void DrawIndirect(BufferBinding arguments) => + Calls.Add($"draw_indirect {arguments.Handle:X}:{arguments.Offset:X}"); + public void Dispatch(uint groupsX, uint groupsY, uint groupsZ) => Calls.Add($"dispatch {groupsX} {groupsY} {groupsZ}"); public bool TryDispatchIndirect(ulong argumentsAddress) => false; @@ -497,6 +538,7 @@ internal static class RenderExecutorFixtures public const ulong ComputeShader = 0x2_0002_0000; public const uint PrimitiveTriangleList = 4; public const uint PrimitiveTriangleStrip = 6; + public const uint PrimitiveTriangleFan = 5; private static Exception Fatal(string message) => new RenderExecutorFatalException(message); diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Rendering/TargetResolverTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Rendering/TargetResolverTests.cs index 90fff02..ec7eb97 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Rendering/TargetResolverTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Rendering/TargetResolverTests.cs @@ -87,6 +87,13 @@ public sealed class TargetResolverTests : IClassFixture Assert.Equal(state.FrontOperations, state.BackOperations); Assert.Equal(state.FrontMasks, state.BackMasks); + // An ALWAYS test ignores the reference, so it carries the replacement value. + context.StencilMask.OperationValue = 0x20; + Assert.Equal(new StencilMasks(0xFF, 0xFF, 0x20), DepthTargetResolver.ResolveState(context, true, Fatal).FrontMasks); + + // A test that reads the reference cannot also carry a different replacement value. + var tested = StencilContext(pass: 4, writeMask: 0xFF, operationValue: 0x20, DepthControl(CompareOp.Less, CompareOp.Equal)); + Assert.Contains("replacement", Assert.Throws(() => DepthTargetResolver.ResolveState(tested, true, Fatal)).Message); context.StencilMask.OperationValue = 0x20; state = DepthTargetResolver.ResolveState(context, true, Fatal); Assert.Equal(0x20u, state.FrontMasks.Reference); // Always does not consume the comparison reference. @@ -101,29 +108,19 @@ public sealed class TargetResolverTests : IClassFixture } [Fact] - public void DepthState_PlayroomReplacementClearsOnlyTheWrittenBit() + public void DepthState_ReplacementWithZeroWrittenBitsUsesZeroOperation() { - var context = StencilContext(pass: 3, writeMask: 0x80, operationValue: 1, depthControl: 0x007007B1); - context.StencilControl = new StencilControlRegisters - { - Fail = 3, Pass = 3, DepthFail = 3, - FailBack = 4, PassBack = 4, DepthFailBack = 4, - }; + var context = StencilContext(pass: 4, writeMask: 0x80, operationValue: 1, + DepthControl(CompareOp.Less, CompareOp.Always)); context.StencilMask.TestValue = 0x80; - context.StencilMask.TestValueBack = 0x80; - context.StencilMask.MaskBack = 0xFF; - context.StencilMask.WriteMaskBack = 0x80; - context.StencilMask.OperationValueBack = 1; var state = DepthTargetResolver.ResolveState(context, true, Fatal); - Assert.Equal(new StencilOperations(StencilOp.Replace, StencilOp.Replace, StencilOp.Replace, CompareOp.Always), state.FrontOperations); - Assert.Equal(new StencilOperations(StencilOp.Zero, StencilOp.Zero, StencilOp.Zero, CompareOp.Always), state.BackOperations); - Assert.Equal(new StencilMasks(0xFF, 0x80, 0x80), state.FrontMasks); - Assert.Equal(state.FrontMasks, state.BackMasks); + Assert.Equal(StencilOp.Zero, state.FrontOperations.PassOperation); + Assert.Equal(0x80u, state.FrontMasks.WriteMask); for (uint oldValue = 0; oldValue <= 0xFF; oldValue++) { - Assert.Equal((oldValue & ~0x80u) | (1u & 0x80u), oldValue & ~state.BackMasks.WriteMask); + Assert.Equal((oldValue & ~0x80u) | (1u & 0x80u), oldValue & ~state.FrontMasks.WriteMask); } } @@ -131,6 +128,23 @@ public sealed class TargetResolverTests : IClassFixture [InlineData(0x0F, 0xF0, 0x20, 0x20)] [InlineData(0xFF, 0xF0, 0x11, 0x10)] public void DepthState_ReplacementMergesOnlyBitsUnusedByComparison(byte compareMask, byte writeMask, byte operationValue, uint reference) + { + var context = StencilContext(pass: 4, writeMask, operationValue, + DepthControl(CompareOp.Less, CompareOp.Equal)); + context.StencilMask.Mask = compareMask; + + var state = DepthTargetResolver.ResolveState(context, true, Fatal); + + Assert.Equal(StencilOp.Replace, state.FrontOperations.PassOperation); + Assert.Equal(reference, state.FrontMasks.Reference); + Assert.Equal((uint)(context.StencilMask.TestValue & compareMask), state.FrontMasks.Reference & compareMask); + Assert.Equal((uint)(operationValue & writeMask), state.FrontMasks.Reference & writeMask); + } + + [Theory] + [InlineData(0x0F, 0xF0, 0x20, 0x20)] + [InlineData(0xFF, 0xF0, 0x11, 0x10)] + public void DepthState_ReplacementMergesOnlyBitsUnusedByComparison_Upstream(byte compareMask, byte writeMask, byte operationValue, uint reference) { var context = StencilContext(pass: 4, writeMask, operationValue, DepthControl(CompareOp.Less, CompareOp.Equal)); context.StencilMask.Mask = compareMask; @@ -143,6 +157,62 @@ public sealed class TargetResolverTests : IClassFixture Assert.Equal((uint)(operationValue & writeMask), state.FrontMasks.Reference & writeMask); } + [Fact] + public void DepthState_EqualTestCanTranslateOperationValueReplacementToMaskedInvert() + { + var context = StencilContext(pass: 4, writeMask: 0xFF, operationValue: 0x40, DepthControl(CompareOp.Less, CompareOp.Equal)); + context.StencilControl.DepthFail = 0; + context.StencilMask.TestValue = 0xFF; + + var state = DepthTargetResolver.ResolveState(context, hasStencil: true, Fatal); + + Assert.Equal(new StencilOperations(StencilOp.Keep, StencilOp.Invert, StencilOp.Keep, CompareOp.Equal), state.FrontOperations); + Assert.Equal(new StencilMasks(0xFF, 0xBF, 0xFF), state.FrontMasks); + + // The Equal test only passes for 0xFF. Invert toggles the differing bits and + // leaves the shared bit intact, exactly matching the guest replacement. + for (var stencil = 0; stencil <= byte.MaxValue; stencil++) + { + var before = (byte)stencil; + var guestAfter = before == 0xFF ? (byte)0x40 : before; + var vulkanAfter = before == state.FrontMasks.Reference + ? (byte)((before & ~(byte)state.FrontMasks.WriteMask) | ((byte)~before & (byte)state.FrontMasks.WriteMask)) + : before; + Assert.Equal(guestAfter, vulkanAfter); + } + + // A replacement in the stencil-fail path has no known input value, and an + // incomplete compare mask does not establish all written bits. + context.StencilControl.Fail = 4; + Assert.Contains("replacement", Assert.Throws(() => DepthTargetResolver.ResolveState(context, true, Fatal)).Message); + context.StencilControl.Fail = 0; + context.StencilMask.Mask = 0x7F; + Assert.Contains("replacement", Assert.Throws(() => DepthTargetResolver.ResolveState(context, true, Fatal)).Message); + } + + [Theory] + [InlineData(0xA5, 0x5A, 0x0F, 0x0F)] + [InlineData(0x00, 0x80, 0xF0, 0x80)] + [InlineData(0xFF, 0xF0, 0x0F, 0x0F)] + [InlineData(0x55, 0xAA, 0xFF, 0x33)] + public void DepthState_EqualReplacementPreservesEveryStencilValue(int test, int replacement, int compareMask, int writeMask) + { + var context = StencilContext(4, (byte)writeMask, (byte)replacement, DepthControl(CompareOp.Less, CompareOp.Equal)); + context.StencilControl.DepthFail = 0; + context.StencilMask.TestValue = (byte)test; + context.StencilMask.Mask = (byte)compareMask; + var state = DepthTargetResolver.ResolveState(context, true, Fatal); + Assert.Equal(StencilOp.Invert, state.FrontOperations.PassOperation); + for (var before = 0; before <= byte.MaxValue; before++) + { + var passes = (before & compareMask) == (test & compareMask); + var expected = passes ? (before & ~writeMask) | (replacement & writeMask) : before; + var hostMask = (byte)state.FrontMasks.WriteMask; + var actual = passes ? (before & ~hostMask) | (~before & hostMask) : before; + Assert.Equal(expected, actual); + } + } + [Fact] public void DepthState_StencilClearDisablesTheOperationsAndBackFaceDecodesSeparately() { @@ -234,6 +304,25 @@ public sealed class TargetResolverTests : IClassFixture Assert.Equal(ImageLayout.DepthAttachmentStencilReadOnlyOptimal, depthOnlyWrite.AttachmentLayout(Format.D32SfloatS8Uint)); } + [Fact] + public void AttachmentLayoutFor_KeepsTheReadOnlyLayoutForDrawsThatOnlyTest() + { + var context = StencilContext(pass: 4, writeMask: 0xFF, operationValue: 0x10, DepthControl(CompareOp.Less, CompareOp.Always)); + var writer = DepthTargetResolver.ResolveState(context, true, Fatal) with { StencilTestEnabled = false }; + var tester = writer with { DepthWriteEnabled = false }; + const Format format = Format.D32SfloatS8Uint; + var readOnly = ImageLayout.DepthStencilReadOnlyOptimal; + + // A sampled attachment is read-only whatever it was in before. + Assert.Equal(readOnly, DepthStencilState.AttachmentLayoutFor(tester, format, sampled: true, ImageLayout.DepthStencilAttachmentOptimal)); + // A test-only draw after a sampling draw stays read-only, so the scope continues. + Assert.Equal(readOnly, DepthStencilState.AttachmentLayoutFor(tester, format, sampled: false, readOnly)); + // Otherwise the writable layout keeps toggling depth writes inside one scope. + Assert.Equal(ImageLayout.DepthStencilAttachmentOptimal, DepthStencilState.AttachmentLayoutFor(tester, format, sampled: false, ImageLayout.DepthStencilAttachmentOptimal)); + Assert.Equal(ImageLayout.DepthStencilAttachmentOptimal, DepthStencilState.AttachmentLayoutFor(tester, format, sampled: false, null)); + Assert.Equal(ImageLayout.DepthStencilAttachmentOptimal, DepthStencilState.AttachmentLayoutFor(writer, format, sampled: false, readOnly)); + } + [Fact] public void DepthResolve_UsesTheBuilderThenAddsTheState() { @@ -252,4 +341,31 @@ public sealed class TargetResolverTests : IClassFixture context.DepthTarget = RegisterWords.Depth(Base, 64, 64, depthTest: false, depthWrite: false); Assert.Null(DepthTargetResolver.Resolve(context, _vulkan.DeviceInfo, Fatal)); } + + [Fact] + public void DepthState_PlayroomReplacementClearsOnlyTheWrittenBit() + { + var context = StencilContext(pass: 3, writeMask: 0x80, operationValue: 1, depthControl: 0x007007B1); + context.StencilControl = new StencilControlRegisters + { + Fail = 3, Pass = 3, DepthFail = 3, + FailBack = 4, PassBack = 4, DepthFailBack = 4, + }; + context.StencilMask.TestValue = 0x80; + context.StencilMask.TestValueBack = 0x80; + context.StencilMask.MaskBack = 0xFF; + context.StencilMask.WriteMaskBack = 0x80; + context.StencilMask.OperationValueBack = 1; + + var state = DepthTargetResolver.ResolveState(context, true, Fatal); + + Assert.Equal(new StencilOperations(StencilOp.Replace, StencilOp.Replace, StencilOp.Replace, CompareOp.Always), state.FrontOperations); + Assert.Equal(new StencilOperations(StencilOp.Zero, StencilOp.Zero, StencilOp.Zero, CompareOp.Always), state.BackOperations); + Assert.Equal(new StencilMasks(0xFF, 0x80, 0x80), state.FrontMasks); + Assert.Equal(state.FrontMasks, state.BackMasks); + for (uint oldValue = 0; oldValue <= 0xFF; oldValue++) + { + Assert.Equal((oldValue & ~0x80u) | (1u & 0x80u), oldValue & ~state.BackMasks.WriteMask); + } + } } diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Scheduling/PresenterSubmissionTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Scheduling/PresenterSubmissionTests.cs index cf6b8b9..47d4829 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Scheduling/PresenterSubmissionTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Scheduling/PresenterSubmissionTests.cs @@ -457,6 +457,8 @@ public sealed class PresenterSubmissionTests PresenterType.GetMethod("PrepareGuestSubmission", InstanceMembers)!.CreateDelegate>(presenter), PresenterType.GetMethod("CompleteGuestSubmission", InstanceMembers)!.CreateDelegate>(presenter)); Set(presenter, "_scheduler", scheduler); + // Each collection below must observe the device state it follows, not a polled value. + scheduler.TimelinePollInterval = 0; scheduler.Begin(new SubmissionContext { QueueName = "test.queue", SubmissionId = 7 }); switch (route) diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Scheduling/TickOwnedResourcesTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Scheduling/TickOwnedResourcesTests.cs new file mode 100644 index 0000000..fe5b03f --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Gpu/Scheduling/TickOwnedResourcesTests.cs @@ -0,0 +1,49 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.Libs.Gpu.Scheduling; +using Xunit; + +namespace SharpEmu.Libs.Tests.Gpu.Scheduling; + +public sealed class TickOwnedResourcesTests +{ + private sealed class Feedback : IDisposable + { + public int Value; + public int Disposals; + public void Dispose() => Disposals++; + } + + [Fact] + public void CompletingEarlierTick_DoesNotConsumeOrResetLaterFeedback() + { + using var resources = new TickOwnedResources(); + var first = resources.Acquire(10, () => new Feedback()); + Assert.Same(first, resources.Acquire(10, () => throw new InvalidOperationException())); + var second = resources.Acquire(11, () => new Feedback()); + Assert.NotSame(first, second); + first.Value = 123; + second.Value = 456; + resources.Complete(10, feedback => + { + Assert.Equal(123, feedback.Value); + feedback.Value = 0; + }); + Assert.Equal(1, first.Disposals); + Assert.Equal(456, second.Value); + Assert.Equal(0, second.Disposals); + resources.Complete(10, _ => throw new InvalidOperationException()); + resources.Complete(11, feedback => Assert.Equal(456, feedback.Value)); + Assert.Equal(1, second.Disposals); + } + + [Fact] + public void FailedRead_StillReleasesResource() + { + using var resources = new TickOwnedResources(); + var feedback = resources.Acquire(1, () => new Feedback()); + Assert.Throws(() => resources.Complete(1, _ => throw new InvalidOperationException())); + Assert.Equal(1, feedback.Disposals); + } +} diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/BitReplicateDeviceTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/BitReplicateDeviceTests.cs new file mode 100644 index 0000000..bb0bbb8 --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/BitReplicateDeviceTests.cs @@ -0,0 +1,54 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.Libs.Gpu.Buffers; +using SharpEmu.Libs.Tests.Gpu.Images; +using SharpEmu.ShaderCompiler; +using SharpEmu.ShaderCompiler.Resources; +using SharpEmu.ShaderCompiler.Tests; +using SharpEmu.ShaderCompiler.Vulkan; +using Xunit; +using static SharpEmu.ShaderCompiler.Tests.Resources.ResourceTestProgram; + +namespace SharpEmu.Libs.Tests.Gpu.Vulkan; + +// S_BITREPLICATE_B64_B32 doubles each source bit (bit i -> bits 2i and 2i+1). Ghost of Yotei's +// bloom blur builds its halo exec mask from 0x003F003F (lanes 0-11 and 32-43); copying the dword +// into both halves left halo LDS unwritten and turned the bloom, then the whole frame, to NaN. +public sealed class BitReplicateDeviceTests(HeadlessVulkanFixture fixture) : IClassFixture +{ + [Theory] + [InlineData(0x003F_003Fu, 0x0000_0FFFu, 0x0000_0FFFu)] + [InlineData(0x8000_0001u, 0x0000_0003u, 0xC000_0000u)] + [InlineData(0x0000_A5C3u, 0xCC33_F00Fu, 0x0000_0000u)] + public void EachBitIsDoubled(uint source, uint low, uint high) + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) return; + var program = Program( + Sop1(0, "SBitreplicateB64B32", 20, Gen5Operand.Scalar(8)), + MoveVectorFromScalar(4, 20, 20), + MoveVectorFromScalar(8, 21, 21), + Vop2(12, "VAndB32", 1, Operand(0), Gen5Operand.Vector(1)), + BufferAccess(16, "BufferStoreDwordx2", 4, dwords: 2, vectorData: 20, offsetEnabled: true, vectorAddress: 1), + EndProgram(24)); + var (plan, resources, layout) = Prepare(program); + var request = new ShaderCompileRequest(plan, resources, layout) { LocalSizeX = 64 }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + using var harness = new ImageTestHarness(vulkan); + using var runner = new LayoutComputeRunner(harness, request, shader.Spirv); + var output = runner.CreateBuffer(8); + var bindings = new Dictionary + { + [DescriptorBindingKind.Buffers] = Enumerable.Repeat(output, Math.Max(1, plan.Info.Buffers.Count)).ToArray(), + }; + var registers = new uint[256]; + registers[6] = 8; + registers[8] = source; + harness.Run(() => runner.Dispatch(registers, bindings, 1)); + harness.AssertNoValidationMessages(); + var bytes = runner.ReadBack(output, 0, 8); + Assert.Equal(low, BitConverter.ToUInt32(bytes, 0)); + Assert.Equal(high, BitConverter.ToUInt32(bytes, 4)); + } +} diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/DeviceAddressShaderTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/DeviceAddressShaderTests.cs index fa8a3b1..0521b24 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/DeviceAddressShaderTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/DeviceAddressShaderTests.cs @@ -92,6 +92,37 @@ public sealed class DeviceAddressShaderTests(HeadlessVulkanFixture fixture, ITes run.Finish(output, nameof(MissingPagesSharingOneBitmapWord_AllRecordFaults)); } + // An invocation reports the missing pages of one fault-bitmap word when it ends. A page + // in another word is reported by a later dispatch, once the first word's pages are mapped. + [Fact] + public void MissingPagesInTwoBitmapWords_AreReportedOneWordPerDispatch() + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) + { + return; + } + + // v1 = load(GuestBase), then v2 = load(GuestBase + 32 pages): the next bitmap word. + var program = Program( + Vop2(0, "VAddU32", OffsetRegister, Operand(0), Gen5Operand.Vector(0)), + GlobalMemory(4, "GlobalLoadDword", AddressLow, OffsetRegister, 1, 1), + Vop2(12, "VAddU32", 6, Operand(32), Gen5Operand.Vector(0)), + Vop2(16, "VLshlrevB32", OffsetRegister, Operand(Gen5SpirvTranslator.DeviceAddressPageBits), Gen5Operand.Vector(6)), + GlobalMemory(20, "GlobalLoadDword", AddressLow, OffsetRegister, 2, 1), + EndProgram(28)); + var far = GuestBase + 32 * PageSize; + var run = new Run(vulkan, program); + run.Dispatch(GuestBase); + Assert.Equal(1u << (int)((GuestBase >> Gen5SpirvTranslator.DeviceAddressPageBits) & 31), run.FaultWord(GuestBase)); + Assert.Equal(0u, run.FaultWord(far)); + + run.MapPage(GuestBase, Pattern((int)PageSize, seed: 51)); + run.Dispatch(GuestBase); + Assert.Equal(1u << (int)((far >> Gen5SpirvTranslator.DeviceAddressPageBits) & 31), run.FaultWord(far)); + run.Finish(output, nameof(MissingPagesInTwoBitmapWords_AreReportedOneWordPerDispatch)); + } + [Fact] public void AddressAboveTheTable_ReadsZeroWithoutTouchingEitherBuffer() { @@ -299,8 +330,536 @@ public sealed class DeviceAddressShaderTests(HeadlessVulkanFixture fixture, ITes lands.Finish(output, nameof(WriteAddressNearTheTopOfTheLowSpace_DoesNotWrap)); } + // A formatted load through a V# the program picks at run time reads its element + // window through the page table: across a page boundary, from an unmapped page, and + // cut short by num_records. + [Fact] + public void RuntimeFormattedLoad_ReadsItsElementThroughThePageTable() + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) + { + return; + } + + const uint Uint32x4 = 75; + var element = GuestBase + PageSize - 8; + + var both = FormatLoadRun(vulkan); + var first = both.MapPage(GuestBase, Pattern((int)PageSize, seed: 11)); + var second = both.MapPage(GuestBase + PageSize, Pattern(64, seed: 12)); + both.MapPage(DescriptorPage, BufferDescriptorBytes(GuestBase, (uint)(PageSize + 64), Uint32x4)); + both.Dispatch(DescriptorPage, extraRegisters: new Dictionary { [13] = (uint)(element - GuestBase) }); + Assert.Equal(ReadWord(first, (int)PageSize - 8), both.ResultWord(0)); + Assert.Equal(ReadWord(first, (int)PageSize - 4), both.ResultWord(4)); + Assert.Equal(ReadWord(second, 0), both.ResultWord(8)); + Assert.Equal(ReadWord(second, 4), both.ResultWord(12)); + + var missingSecond = FormatLoadRun(vulkan); + var only = missingSecond.MapPage(GuestBase, Pattern((int)PageSize, seed: 13)); + missingSecond.MapPage(DescriptorPage, BufferDescriptorBytes(GuestBase, (uint)(PageSize + 64), Uint32x4)); + missingSecond.Dispatch(DescriptorPage, extraRegisters: new Dictionary { [13] = (uint)(element - GuestBase) }); + Assert.Equal(ReadWord(only, (int)PageSize - 8), missingSecond.ResultWord(0)); + Assert.Equal(ReadWord(only, (int)PageSize - 4), missingSecond.ResultWord(4)); + Assert.Equal(0u, missingSecond.ResultWord(8)); + Assert.Equal(0u, missingSecond.ResultWord(12)); + Assert.NotEqual(0u, missingSecond.FaultWord(GuestBase + PageSize)); + + // num_records ends inside the element: the whole element reads as zero, and the + // unmapped page past the end is never walked, so it records no fault. + var cut = FormatLoadRun(vulkan); + cut.MapPage(GuestBase, Pattern((int)PageSize, seed: 14)); + cut.MapPage(DescriptorPage, BufferDescriptorBytes(GuestBase, (uint)(PageSize - 4), Uint32x4)); + cut.Dispatch(DescriptorPage, extraRegisters: new Dictionary { [13] = (uint)(element - GuestBase) }); + for (uint component = 0; component < 4; component++) + { + Assert.Equal(0u, cut.ResultWord(component * 4)); + } + + Assert.Equal(0u, cut.FaultWord(GuestBase + PageSize)); + + // 8_8_8_8 UNORM inside one page: each byte converts to byte / 255 (within one ulp; + // the conversion multiplies by 1/255). + const uint Unorm8x4 = 56; + var unorm = FormatLoadRun(vulkan); + var bytes = unorm.MapPage(GuestBase, Pattern((int)PageSize, seed: 15)); + unorm.MapPage(DescriptorPage, BufferDescriptorBytes(GuestBase, (uint)(PageSize), Unorm8x4)); + unorm.Dispatch(DescriptorPage, extraRegisters: new Dictionary { [13] = 36 }); + for (var component = 0; component < 4; component++) + { + var expected = BitConverter.SingleToUInt32Bits(bytes[36 + component] / 255f); + Assert.InRange(unorm.ResultWord((uint)component * 4), expected - 1, expected + 1); + } + + foreach (var run in new[] { both, missingSecond, cut, unorm }) + { + run.Finish(output, nameof(RuntimeFormattedLoad_ReadsItsElementThroughThePageTable)); + } + } + // ---- programs ---- + [Theory] + [InlineData(false, true, true)] + [InlineData(false, true, false)] + [InlineData(false, false, true)] + [InlineData(false, false, false)] + [InlineData(true, true, true)] + [InlineData(true, true, false)] + [InlineData(true, false, true)] + [InlineData(true, false, false)] + public void GpuLoadedDescriptor_MultipleWordsRespectEachPage(bool formatted, bool mapFirst, bool mapSecond) + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) return; + var run = new Run(vulkan, GpuDescriptorLoadProgram(formatted)); + Assert.Contains(run.Request.Memory.Entries, memory => memory.DeviceDescriptor); + var first = Pattern((int)PageSize, seed: 31); + var second = Pattern((int)PageSize, seed: 32); + if (mapFirst) run.MapPage(GuestBase, first); + if (mapSecond) run.MapPage(GuestBase + PageSize, second); + var descriptorPage = GuestBase + 4 * PageSize; + run.MapPage(descriptorPage, BufferDescriptorBytes(GuestBase, (uint)PageSize + 16)); + run.Dispatch(descriptorPage, extraRegisters: new Dictionary { [13] = (uint)PageSize - 8 }); + Assert.Equal(mapFirst ? ReadWord(first, (int)PageSize - 8) : 0u, run.ResultWord(0)); + Assert.Equal(mapFirst ? ReadWord(first, (int)PageSize - 4) : 0u, run.ResultWord(4)); + Assert.Equal(mapSecond ? ReadWord(second, 0) : 0u, run.ResultWord(8)); + Assert.Equal(mapSecond ? ReadWord(second, 4) : 0u, run.ResultWord(12)); + var firstBit = 1u << (int)((GuestBase >> Gen5SpirvTranslator.DeviceAddressPageBits) & 31); + var secondBit = firstBit << 1; + Assert.Equal((mapFirst ? 0u : firstBit) | (mapSecond ? 0u : secondBit), run.FaultWord(GuestBase)); + run.Finish(output, nameof(GpuLoadedDescriptor_MultipleWordsRespectEachPage)); + } + + [Theory] + [InlineData(false, false)] + [InlineData(false, true)] + [InlineData(true, false)] + [InlineData(true, true)] + public void GpuLoadedDescriptor_BoundsAndOffsetWrapDoNotWalkSkippedWords(bool formatted, bool wrapOffset) + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) return; + var run = new Run(vulkan, GpuDescriptorLoadProgram(formatted)); + Assert.Contains(run.Request.Memory.Entries, memory => memory.DeviceDescriptor); + var bytes = run.MapPage(GuestBase, Pattern((int)PageSize, seed: 33)); + var descriptorPage = GuestBase + 4 * PageSize; + var size = wrapOffset ? 4u : (uint)PageSize - 4; + run.MapPage(descriptorPage, BufferDescriptorBytes(GuestBase, size)); + var offset = wrapOffset ? 0xFFFF_FFFCu : (uint)PageSize - 8; + run.Dispatch(descriptorPage, extraRegisters: new Dictionary { [13] = offset }); + for (uint component = 0; component < 4; component++) + { + var expected = formatted ? 0u : wrapOffset + ? component == 1 ? ReadWord(bytes, 0) : 0u + : component == 0 ? ReadWord(bytes, (int)PageSize - 8) : 0u; + Assert.Equal(expected, run.ResultWord(component * 4)); + } + Assert.All(run.FaultWords(), word => Assert.Equal(0u, word)); + run.Finish(output, nameof(GpuLoadedDescriptor_BoundsAndOffsetWrapDoNotWalkSkippedWords)); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void GpuLoadedDescriptor_GuestAddressWrapResolvesTheNewPage(bool formatted) + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) return; + var run = new Run(vulkan, GpuDescriptorLoadProgram(formatted)); + var bytes = run.MapPage(0, Pattern((int)PageSize, seed: 34)); + var descriptorPage = GuestBase + 4 * PageSize; + run.MapPage(descriptorPage, BufferDescriptorBytes(0x0000_FFFF_FFFF_FFFCu, 16)); + run.Dispatch(descriptorPage, extraRegisters: new Dictionary { [13] = 0 }); + Assert.Equal(0u, run.ResultWord(0)); + for (uint component = 1; component < 4; component++) + Assert.Equal(ReadWord(bytes, (int)(component - 1) * 4), run.ResultWord(component * 4)); + Assert.All(run.FaultWords(), word => Assert.Equal(0u, word)); + run.Finish(output, nameof(GpuLoadedDescriptor_GuestAddressWrapResolvesTheNewPage)); + } + + public static IEnumerable PartialFormats() + { + for (uint format = 1; format < 128; format++) + { + if (!Gfx10UnifiedFormat.TryDecode(format, out _, out _)) continue; + for (uint count = 1; count < 4; count++) + yield return [format, count]; + } + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void GpuLoadedDescriptor_OffsetAndGuestWrapKeepTheMiddlePage(bool formatted) + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) return; + var run = new Run(vulkan, GpuDescriptorLoadProgram(formatted, formatted ? 1u : 4u)); + var data = run.MapPage(0, Pattern(64, seed: 35)); + var descriptorPage = GuestBase + 4 * PageSize; + var descriptor = BufferDescriptorBytes(0x0000_FFFF_0000_0008ul, uint.MaxValue); + if (formatted) WriteWord(descriptor, 12, 5u | (75u << 12)); // X reads the Y component. + run.MapPage(descriptorPage, descriptor); + run.Dispatch(descriptorPage, extraRegisters: new Dictionary { [13] = 0xFFFF_FFF4u }); + // The first guest page is at the end of the 48-bit space, the middle + // page wraps to zero, and the last page follows the uint32 offset wrap. + Assert.Equal(formatted ? ReadWord(data, 0) : 0u, run.ResultWord(0)); + if (!formatted) + { + Assert.Equal(ReadWord(data, 0), run.ResultWord(4)); + Assert.Equal(0u, run.ResultWord(8)); // Last dword exceeds num_records. + Assert.Equal(0u, run.ResultWord(12)); + } + Assert.All(run.FaultWords(), word => Assert.Equal(0u, word)); + run.Finish(output, nameof(GpuLoadedDescriptor_OffsetAndGuestWrapKeepTheMiddlePage)); + } + + [Theory] + [MemberData(nameof(PartialFormats))] + public void GpuLoadedDescriptor_PartialFormatMatchesFullLoad(uint format, uint count) + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) return; + var data = Pattern(64, seed: format); + for (uint selector = 0; selector < 8; selector++) + { + uint swizzle = 0; + for (uint component = 0; component < count; component++) + swizzle |= ((selector + component) & 7) << (int)(component * 3); + foreach (uint offset in new uint[] { 1, 60 }) + { + uint[] Execute(uint width) + { + var run = new Run(vulkan, GpuDescriptorLoadProgram(true, width)); + Assert.Contains(run.Request.Memory.Entries, memory => memory.Pc == 36 && memory.DeviceDescriptor); + run.MapPage(GuestBase, data); + var descriptorPage = GuestBase + 4 * PageSize; + var descriptor = BufferDescriptorBytes(GuestBase, 64); + WriteWord(descriptor, 12, swizzle | (format << 12)); + run.MapPage(descriptorPage, descriptor); + run.Dispatch(descriptorPage, extraRegisters: new Dictionary { [13] = offset }); + var result = Enumerable.Range(0, (int)count).Select(index => run.ResultWord((uint)index * 4)).ToArray(); + Assert.All(run.FaultWords(), word => Assert.Equal(0u, word)); + run.Finish(output, nameof(GpuLoadedDescriptor_PartialFormatMatchesFullLoad)); + return result; + } + Assert.Equal(Execute(4), Execute(count)); + } + } + } + + [Theory] + [InlineData(75u, 1u)] + [InlineData(75u, 2u)] + [InlineData(75u, 3u)] + [InlineData(75u, 4u)] + [InlineData(76u, 4u)] + [InlineData(77u, 1u)] + [InlineData(77u, 2u)] + [InlineData(77u, 3u)] + [InlineData(77u, 4u)] + public void GpuLoadedD16_PacksComponentsWithoutOverwritingNeighbourRegisters(uint format, uint count) + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) return; + var load = BufferAccess(52, count switch + { + 1 => "BufferLoadFormatD16X", 2 => "BufferLoadFormatD16Xy", + 3 => "BufferLoadFormatD16Xyz", _ => "BufferLoadFormatD16Xyzw", + }, 16, dwords: (count + 1) / 2, vectorData: 4); + load = load with { Control = ((Gen5BufferMemoryControl)load.Control!) with + { PackedD16 = true, FormatComponentCount = count } }; + var program = Program([ + .. GpuDescriptorLoadProgram(true).Instructions.Take(8), + MoveVector(36, 6, 0xAABBCCDD), MoveVector(44, 7, 0x11223344), + load, BufferAccess(60, "BufferStoreDwordx4", ResultRegister, dwords: 4, vectorData: 4), EndProgram(68), + ]); + var run = new Run(vulkan, program); + var data = new byte[64]; + uint[] components = format == 77 + ? [0x3F800000u, 0xBF000000u, 0x40400000u, 0x40800000u] + : [0x00011234u, 0xFFFF8000u, 0x0002ABCDu, 0xFFFF7FFFu]; + for (var i = 0; i < 4; i++) WriteWord(data, i * 4, components[i]); + run.MapPage(GuestBase, data); + var descriptorPage = GuestBase + 4 * PageSize; + run.MapPage(descriptorPage, BufferDescriptorBytes(GuestBase, 64, format)); + run.Dispatch(descriptorPage); + for (uint component = 0; component < count; component++) + { + var expected = format == 77 + ? BitConverter.HalfToUInt16Bits((Half)BitConverter.UInt32BitsToSingle(components[component])) + : components[component] & 0xFFFF; + var actual = (run.ResultWord(component / 2 * 4) >> (int)((component & 1) * 16)) & 0xFFFF; + Assert.Equal(expected, actual); + } + Assert.Equal(0xAABBCCDDu, run.ResultWord(8)); + Assert.Equal(0x11223344u, run.ResultWord(12)); + run.Finish(output, nameof(GpuLoadedD16_PacksComponentsWithoutOverwritingNeighbourRegisters)); + } + + [Fact] + public void RuntimeFormatWithoutCandidates_ReadsTheActualGuestDescriptor() + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) return; + var run = new Run(vulkan, GpuDescriptorLoadProgram(true), configure: request => + { + Assert.True(request.Memory.TryGetIndex(36, 0, out var index)); + Assert.False(request.BufferCandidateTableByMemoryIndex.ContainsKey(index)); + request.Memory[index].DeviceDescriptor = false; + request.Memory[index].BufferDescriptor = new GuestBufferDescriptor { Provenance = BufferDescriptorProvenance.Runtime }; + }); + var data = new byte[64]; + for (var component = 0; component < 4; component++) + WriteWord(data, component * 4, BitConverter.SingleToUInt32Bits(component + 1f)); + run.MapPage(GuestBase, data); + var descriptorPage = GuestBase + 4 * PageSize; + run.MapPage(descriptorPage, BufferDescriptorBytes(GuestBase, 64, 77)); + run.Dispatch(descriptorPage); + for (var component = 0; component < 4; component++) + Assert.Equal(ReadWord(data, component * 4), run.ResultWord((uint)component * 4)); + run.Finish(output, nameof(RuntimeFormatWithoutCandidates_ReadsTheActualGuestDescriptor)); + } + + [Theory] + [InlineData(75u)] + [InlineData(76u)] + [InlineData(77u)] + public void GpuLoadedD16Store_UnpacksBothRegistersIntoFourComponents(uint format) + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) return; + var store = BufferAccess(52, "BufferStoreFormatD16Xyzw", 16, dwords: 2, vectorData: 4); + store = store with { Control = ((Gen5BufferMemoryControl)store.Control!) with + { PackedD16 = true, FormatComponentCount = 4 } }; + var program = Program([.. GpuDescriptorStoreProgram().Instructions.Take(11), store, EndProgram(60)]); + var run = new Run(vulkan, program); + run.MapPage(GuestBase, new byte[64]); + var descriptorPage = GuestBase + 4 * PageSize; + run.MapPage(descriptorPage, BufferDescriptorBytes(GuestBase, 64, format)); + ushort[] components = format == 77 ? [0x3C00, 0xB800, 0x4200, 0x4400] : [0x1234, 0x8000, 0xABCD, 0x7FFF]; + run.Dispatch(descriptorPage, extraRegisters: new Dictionary + { + [24] = components[0] | ((uint)components[1] << 16), + [25] = components[2] | ((uint)components[3] << 16), + [26] = 0xDEADBEEF, [27] = 0xBAD0CAFE, + }); + for (uint component = 0; component < 4; component++) + { + var expected = format switch + { + 77 => BitConverter.SingleToUInt32Bits((float)BitConverter.UInt16BitsToHalf(components[component])), + 76 => unchecked((uint)(int)(short)components[component]), + _ => components[component], + }; + Assert.Equal(expected, run.PageWord(GuestBase, component * 4)); + } + run.Finish(output, nameof(GpuLoadedD16Store_UnpacksBothRegistersIntoFourComponents)); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void SharedFormattedLoad_RecordsTheFaultingCallerPc(bool firstPageMapped) + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) return; + var program = Program([ + .. GpuDescriptorLoadProgram(true, 1).Instructions.Take(8), + BufferAccess(36, "BufferLoadFormatX", 16, dwords: 1, vectorData: 4), + BufferAccess(44, "BufferLoadFormatX", 16, offset: (int)PageSize + 32, dwords: 1, vectorData: 5), + BufferAccess(52, "BufferStoreDwordx2", ResultRegister, dwords: 2, vectorData: 4), EndProgram(60), + ]); + var run = new Run(vulkan, program, traceFaults: true); + if (firstPageMapped) run.MapPage(GuestBase, Pattern((int)PageSize)); + var descriptorPage = GuestBase + 4 * PageSize; + run.MapPage(descriptorPage, BufferDescriptorBytes(GuestBase, (uint)PageSize * 2)); + run.Dispatch(descriptorPage); + Assert.Equal(0u, run.ResultWord(4)); + Assert.Equal(firstPageMapped ? ReadWord(Pattern((int)PageSize), 0) : 0u, run.ResultWord(0)); + var faults = run.FaultWords(); + Assert.Equal(1u, faults[^8]); + Assert.Equal(firstPageMapped ? 44u : 36u, faults[^5]); + run.Finish(output, nameof(SharedFormattedLoad_RecordsTheFaultingCallerPc)); + } + + [Theory] + [InlineData(32u, 75u)] + [InlineData(32u, 77u)] + [InlineData(64u, 75u)] + [InlineData(64u, 77u)] + public void RepeatedD16Loads_KeepPerLaneValuesAndInactiveRegisters(uint waveSize, uint secondFormat) + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) return; + Gen5ShaderInstruction Load(uint pc) => BufferAccess(pc, "BufferLoadFormatD16Xy", 16, + dwords: 1, vectorData: 4, offsetEnabled: true, vectorAddress: OffsetRegister) with + { + Control = new Gen5BufferMemoryControl(1, OffsetRegister, 4, 16, 0, + false, true, false, false, false, 0, true, 2), + }; + var program = Program([ + .. GpuDescriptorLoadProgram(true).Instructions.Take(8), + Vop2(36, "VLshlrevB32", 2, Operand(3), Gen5Operand.Vector(1)), + Vop2(40, "VAddU32", 2, Gen5Operand.Vector(0), Gen5Operand.Vector(2)), + Vop2(44, "VLshlrevB32", OffsetRegister, Operand(3), Gen5Operand.Vector(2)), + Load(48), + ScalarLoad(56, 8, destination: 16, count: 4, immediateOffset: 16), + MoveScalar(64, 20, 0xAAAAAAAA), MoveScalar(72, 21, 0xAAAAAAAA), + Sop1(80, "SMovB64", 126, Gen5Operand.Scalar(20)), + Load(84), + Sop1(92, "SMovB64", 126, Gen5Operand.Source(193)), + Vop2(96, "VLshlrevB32", 5, Operand(2), Gen5Operand.Vector(2)), + BufferAccess(100, "BufferStoreDword", ResultRegister, vectorData: 4, + offsetEnabled: true, vectorAddress: 5), + EndProgram(108), + ]); + var run = new Run(vulkan, program, threadCount: 64, localSizeY: 8, + waveSize: waveSize, enableExecGuardElision: false); + var first = new byte[512]; + var second = new byte[512]; + for (var lane = 0; lane < 64; lane++) + { + WriteWord(first, lane * 8, BitConverter.SingleToUInt32Bits(lane + 1f)); + WriteWord(first, lane * 8 + 4, BitConverter.SingleToUInt32Bits(-lane - 1f)); + WriteWord(second, lane * 8, secondFormat == 77 + ? BitConverter.SingleToUInt32Bits(lane + 65f) : (uint)lane + 1000); + WriteWord(second, lane * 8 + 4, secondFormat == 77 + ? BitConverter.SingleToUInt32Bits(-lane - 65f) : (uint)lane + 2000); + } + run.MapPage(GuestBase, first); + run.MapPage(GuestBase + PageSize, second); + var descriptors = BufferDescriptorBytes(GuestBase, 512, 77) + .Concat(BufferDescriptorBytes(GuestBase + PageSize, 512, secondFormat)).ToArray(); + var descriptorPage = GuestBase + 4 * PageSize; + run.MapPage(descriptorPage, descriptors); + run.Dispatch(descriptorPage); + for (uint lane = 0; lane < 64; lane++) + { + var changed = (lane & 1) != 0; + var integer = changed && secondFormat == 75; + var low = integer ? lane + 1000 : (uint)BitConverter.HalfToUInt16Bits((Half)(lane + (changed ? 65f : 1f))); + var high = integer ? lane + 2000 : (uint)BitConverter.HalfToUInt16Bits((Half)(-lane - (changed ? 65f : 1f))); + Assert.Equal(low | (high << 16), run.ResultWord(lane * 4)); + } + run.Finish(output, nameof(RepeatedD16Loads_KeepPerLaneValuesAndInactiveRegisters)); + } + + private static Gen5ShaderProgram GpuDescriptorLoadProgram(bool formatted, uint count = 4) => Program( + MoveVectorFromScalar(0, 8, AddressLow), + Vop2(4, "VAddU32", 8, Gen5Operand.Vector(8), Gen5Operand.Vector(0)), + ReadFirstLane(8, 8, 8), MoveScalarRegister(12, 9, AddressHigh), + MoveScalar(16, 10, 16), MoveScalar(20, 11, 0), + ScalarBufferLoad(24, 8, destination: 16, count: 4), + MoveVectorFromScalar(32, OffsetRegister, 13), + BufferAccess(36, formatted ? count switch + { 1 => "BufferLoadFormatX", 2 => "BufferLoadFormatXy", 3 => "BufferLoadFormatXyz", _ => "BufferLoadFormatXyzw" } + : "BufferLoadDwordx4", 16, + dwords: count, vectorData: 4, offsetEnabled: true, vectorAddress: OffsetRegister), + BufferAccess(44, "BufferStoreDwordx4", ResultRegister, dwords: 4, vectorData: 4), + EndProgram(52)); + + private static byte[] BufferDescriptorBytes(ulong address, uint size, uint format = 75) + { + var bytes = new byte[16]; + WriteWord(bytes, 0, (uint)address); + WriteWord(bytes, 4, (uint)(address >> 32) & 0xFFFF); + WriteWord(bytes, 8, size); + WriteWord(bytes, 12, 4u | (5u << 3) | (6u << 6) | (7u << 9) | (format << 12)); + return bytes; + } + + // A typed store through a V# the GPU reads from memory: the runtime format selects the + // encoding, and the element is written through the page table without touching the + // bytes around it, across a page boundary and at an unaligned offset. An element cut by + // num_records, or a format that encodes nothing, writes nothing. + [Theory] + [InlineData(56u, 2u, 4u, true)] // 8_8_8_8 UNORM, 2 bytes before the page end + [InlineData(75u, 8u, 16u, true)] // 32_32_32_32 UINT, 8 bytes before the page end + [InlineData(75u, 8u, 12u, false)] // num_records ends inside the element + [InlineData(0u, 8u, 16u, false)] // invalid format + public void GpuLoadedDescriptor_FormattedStoreWritesItsElementThroughThePageTable(uint format, uint fromEnd, uint recordBytesPastOffset, bool lands) + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) return; + var run = new Run(vulkan, GpuDescriptorStoreProgram()); + Assert.Contains(run.Request.Memory.Entries, memory => memory.DeviceDescriptor); + var first = run.MapPage(GuestBase, Pattern((int)PageSize, seed: 41)); + var second = run.MapPage(GuestBase + PageSize, Pattern((int)PageSize, seed: 42)); + var descriptorPage = GuestBase + 4 * PageSize; + var offset = (uint)PageSize - fromEnd; + run.MapPage(descriptorPage, BufferDescriptorBytes(GuestBase, offset + recordBytesPastOffset, format)); + uint[] values = format == 56 + ? [BitConverter.SingleToUInt32Bits(1f), BitConverter.SingleToUInt32Bits(0.5f), 0u, BitConverter.SingleToUInt32Bits(1f)] + : [0x1111_1111, 0x2222_2222, 0x3333_3333, 0x4444_4444]; + run.Dispatch(descriptorPage, extraRegisters: new Dictionary + { + [13] = offset, [24] = values[0], [25] = values[1], [26] = values[2], [27] = values[3], + }); + + // The plane around the element, as bytes: the end of the first page and the start of the second. + var expected = new byte[32]; + first.AsSpan((int)PageSize - 16, 16).CopyTo(expected); + second.AsSpan(0, 16).CopyTo(expected.AsSpan(16)); + if (lands) + { + byte[] element = format == 56 + ? [0xFF, 0x80, 0x00, 0xFF] + : [.. values.SelectMany(BitConverter.GetBytes)]; + element.CopyTo(expected.AsSpan(16 - (int)fromEnd)); + } + + var actual = new byte[32]; + for (var word = 0; word < 4; word++) + { + BitConverter.TryWriteBytes(actual.AsSpan(word * 4), run.PageWord(GuestBase, PageSize - 16 + (ulong)word * 4)); + BitConverter.TryWriteBytes(actual.AsSpan(16 + word * 4), run.PageWord(GuestBase + PageSize, (ulong)word * 4)); + } + + Assert.Equal(expected, actual); + run.Finish(output, nameof(GpuLoadedDescriptor_FormattedStoreWritesItsElementThroughThePageTable)); + } + + // s[16:19] = the V# at s[8:9] + v0, read by the GPU; v4..v7 = s24..s27; + // typed_store(s[16:19], offset s13) of v4..v7. + private static Gen5ShaderProgram GpuDescriptorStoreProgram() => Program( + MoveVectorFromScalar(0, 8, AddressLow), + Vop2(4, "VAddU32", 8, Gen5Operand.Vector(8), Gen5Operand.Vector(0)), + ReadFirstLane(8, 8, 8), MoveScalarRegister(12, 9, AddressHigh), + MoveScalar(16, 10, 16), MoveScalar(20, 11, 0), + ScalarBufferLoad(24, 8, destination: 16, count: 4), + MoveVectorFromScalar(32, OffsetRegister, 13), + MoveVectorFromScalar(36, 4, 24), MoveVectorFromScalar(40, 5, 25), + MoveVectorFromScalar(44, 6, 26), MoveVectorFromScalar(48, 7, 27), + BufferAccess(52, "BufferStoreFormatXyzw", 16, dwords: 4, vectorData: 4, offsetEnabled: true, vectorAddress: OffsetRegister), + EndProgram(60)); + + // The V# sits on its own page, which the GPU reads to find the formatted buffer. + private const ulong DescriptorPage = GuestBase + 4 * PageSize; + + // A formatted load reads through the page table only when its V# is unknown until the + // shader runs; a descriptor read by the GPU is one, so the program loads it that way. + private Run FormatLoadRun(HeadlessVulkan vulkan) + { + var run = new Run(vulkan, GpuDescriptorLoadProgram()); + Assert.Contains(run.Request.Memory.Entries, memory => memory.DeviceDescriptor); + return run; + } + + // s[16:19] = the V# at s[8:9] + v0, read by the GPU; v4..v7 = format_load(s[16:19], s13); + // result[0..3] = v4..v7. + private static Gen5ShaderProgram GpuDescriptorLoadProgram() => Program( + MoveVectorFromScalar(0, 8, AddressLow), + Vop2(4, "VAddU32", 8, Gen5Operand.Vector(8), Gen5Operand.Vector(0)), + ReadFirstLane(8, 8, 8), MoveScalarRegister(12, 9, AddressHigh), + MoveScalar(16, 10, 16), MoveScalar(20, 11, 0), + ScalarBufferLoad(24, 8, destination: 16, count: 4), + MoveVectorFromScalar(32, OffsetRegister, 13), + BufferAccess(36, "BufferLoadFormatXyzw", 16, 0, 4, vectorData: 4, offsetEnabled: true, vectorAddress: OffsetRegister), + BufferAccess(44, "BufferStoreDwordx4", ResultRegister, 0, 4, vectorData: 4), + EndProgram(52)); + // v3 = 0; v1 = load(s[0:1] + v3 + offset); result[0] = v1. private static Gen5ShaderProgram LoadProgram(string opcode, int offset) => Program( MoveVector(0, OffsetRegister, 0), @@ -346,11 +905,21 @@ public sealed class DeviceAddressShaderTests(HeadlessVulkanFixture fixture, ITes private readonly ulong _tableEntries; private GpuBuffer? _pageTable; - public Run(HeadlessVulkan vulkan, Gen5ShaderProgram program, uint threadCount = 1, ulong tableEntries = TableEntries) + public Run(HeadlessVulkan vulkan, Gen5ShaderProgram program, uint threadCount = 1, ulong tableEntries = TableEntries, + Action? configure = null, bool traceFaults = false, + uint localSizeY = 1, uint waveSize = 32, bool enableExecGuardElision = true) { var (plan, resources, layout) = Prepare(program); Plan = plan; - Request = new ShaderCompileRequest(plan, resources, layout) { LocalSizeX = threadCount, ThreadCountX = threadCount }; + Request = new ShaderCompileRequest(plan, resources, layout) + { + LocalSizeX = threadCount / localSizeY, LocalSizeY = localSizeY, + ThreadCountX = threadCount / localSizeY, + ThreadCountY = localSizeY == 1 ? ShaderCompileRequest.UnboundedThreadCount : localSizeY, + WaveSize = waveSize, EnableExecGuardElision = enableExecGuardElision, + TraceDeviceAddressFaults = traceFaults, + }; + configure?.Invoke(Request); Assert.True(resources.Info.UsesDeviceAddresses); Assert.True(Gen5SpirvTranslator.TryCompileProgram(Request, out var shader, out var error), error); _harness = new ImageTestHarness(vulkan); @@ -370,10 +939,15 @@ public sealed class DeviceAddressShaderTests(HeadlessVulkanFixture fixture, ITes return bytes; } - public void Dispatch(ulong guestAddress, (ulong Base, ulong Size)? writtenRange = null) + public void Dispatch(ulong guestAddress, (ulong Base, ulong Size)? writtenRange = null, IReadOnlyDictionary? extraRegisters = null) { _pageTable = _runner.CreatePageTable(_tableEntries, _pages.Select(page => (page.Key, page.Value, 0ul))); var registers = new uint[256]; + foreach (var (register, value) in extraRegisters ?? new Dictionary()) + { + registers[register] = value; + } + registers[AddressLow] = (uint)guestAddress; registers[AddressHigh] = (uint)(guestAddress >> 32); registers[ResultRegister + 2] = ResultBytes; diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/DriverShadowMemoryProbeTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/DriverShadowMemoryProbeTests.cs new file mode 100644 index 0000000..00602fd --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/DriverShadowMemoryProbeTests.cs @@ -0,0 +1,151 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Runtime.InteropServices; +using SharpEmu.Libs.Gpu.Buffers; +using SharpEmu.Libs.Tests.Gpu.Images; +using Silk.NET.Vulkan; +using Xunit; +using Xunit.Abstractions; + +namespace SharpEmu.Libs.Tests.Gpu.Vulkan; + +// TEMP DIAG: does the driver commit host write-combined memory for device-local allocations? +public sealed unsafe partial class DriverShadowMemoryProbeTests(HeadlessVulkanFixture fixture, ITestOutputHelper output) : IClassFixture +{ + [StructLayout(LayoutKind.Sequential)] + private struct MemoryBasicInformation + { + public nint BaseAddress; + public nint AllocationBase; + public uint AllocationProtect; + public ushort PartitionId; + public nuint RegionSize; + public uint State; + public uint Protect; + public uint Type; + } + + [LibraryImport("kernel32.dll")] + private static partial nuint VirtualQuery(nint address, out MemoryBasicInformation information, nuint length); + + private static (int Count, ulong Bytes) WriteCombinedPrivate() + { + var count = 0; + ulong bytes = 0; + nint address = 0; + while (VirtualQuery(address, out var info, (nuint)sizeof(MemoryBasicInformation)) != 0) + { + if (info.State == 0x1000 && info.Type == 0x20000 && (info.Protect & 0x400) != 0) + { + count++; + bytes += (ulong)info.RegionSize; + } + + var next = (ulong)info.BaseAddress + (ulong)info.RegionSize; + if (next <= (ulong)address || next >= 0x7FFF_FFFF_0000) break; + address = (nint)next; + } + + return (count, bytes); + } + + [Theory] + [InlineData(32u, true)] + [InlineData(32u, false)] + [InlineData(256u, true)] + public void DeviceLocalAllocationsAndWriteCombinedHostMemory(uint megabytes, bool deviceAddress) + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) return; + using var harness = new ImageTestHarness(vulkan); + var before = WriteCombinedPrivate(); + var buffers = new List(); + var flags = deviceAddress ? GpuBuffer.AllFlags : BufferUsageFlags.StorageBufferBit | BufferUsageFlags.TransferDstBit | BufferUsageFlags.TransferSrcBit; + for (var index = 0; index < 16; index++) + { + buffers.Add(new GpuBuffer(harness.Device, harness.Scheduler, GpuBufferUsage.DeviceLocal, 0, flags, (ulong)megabytes << 20)); + } + + var after = WriteCombinedPrivate(); + foreach (var buffer in buffers) buffer.Dispose(); + var freed = WriteCombinedPrivate(); + output.WriteLine($"size={megabytes}MB deviceAddress={deviceAddress}: wc before n={before.Count} {before.Bytes >> 20}MB, " + + $"after 16 allocations n={after.Count} {after.Bytes >> 20}MB, after free n={freed.Count} {freed.Bytes >> 20}MB"); + } + // A device of its own with the memory-priority and pageable-device-local extensions, then + // raw device-local allocations with and without them. + [Theory] + [InlineData(false, false)] + [InlineData(true, false)] + [InlineData(false, true)] + [InlineData(true, true)] + public void PageableAndPriorityDeviceLocalMemory(bool pageable, bool priority) + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) return; + var vk = vulkan.Vk; + var queuePriority = 1f; + var queueInfo = new DeviceQueueCreateInfo + { + SType = StructureType.DeviceQueueCreateInfo, + QueueFamilyIndex = vulkan.QueueFamily, + QueueCount = 1, + PQueuePriorities = &queuePriority, + }; + var pageableFeatures = new PhysicalDevicePageableDeviceLocalMemoryFeaturesEXT + { + SType = StructureType.PhysicalDevicePageableDeviceLocalMemoryFeaturesExt, + PageableDeviceLocalMemory = pageable, + }; + var priorityFeatures = new PhysicalDeviceMemoryPriorityFeaturesEXT + { + SType = StructureType.PhysicalDeviceMemoryPriorityFeaturesExt, + MemoryPriority = true, + PNext = &pageableFeatures, + }; + var names = new List { "VK_EXT_memory_priority" }; + if (pageable) names.Add("VK_EXT_pageable_device_local_memory"); + var extensions = Silk.NET.Core.Native.SilkMarshal.StringArrayToPtr(names.ToArray()); + var deviceInfo = new DeviceCreateInfo + { + SType = StructureType.DeviceCreateInfo, + PNext = &priorityFeatures, + QueueCreateInfoCount = 1, + PQueueCreateInfos = &queueInfo, + EnabledExtensionCount = (uint)names.Count, + PpEnabledExtensionNames = (byte**)extensions, + }; + Assert.Equal(Result.Success, vk.CreateDevice(vulkan.Physical, &deviceInfo, null, out var device)); + vk.GetPhysicalDeviceMemoryProperties(vulkan.Physical, out var properties); + uint deviceLocal = 0; + for (uint type = 0; type < properties.MemoryTypeCount; type++) + { + if (properties.MemoryTypes[(int)type].PropertyFlags == MemoryPropertyFlags.DeviceLocalBit) { deviceLocal = type; break; } + } + + var workingSetBefore = System.Diagnostics.Process.GetCurrentProcess().WorkingSet64; + var before = WriteCombinedPrivate(); + var memories = new List(); + for (var index = 0; index < 16; index++) + { + var priorityInfo = new MemoryPriorityAllocateInfoEXT { SType = StructureType.MemoryPriorityAllocateInfoExt, Priority = 1f }; + var allocate = new MemoryAllocateInfo + { + SType = StructureType.MemoryAllocateInfo, + PNext = priority ? &priorityInfo : null, + AllocationSize = 32UL << 20, + MemoryTypeIndex = deviceLocal, + }; + Assert.Equal(Result.Success, vk.AllocateMemory(device, &allocate, null, out var memory)); + memories.Add(memory); + } + + var after = WriteCombinedPrivate(); + var workingSetAfter = System.Diagnostics.Process.GetCurrentProcess().WorkingSet64; + foreach (var memory in memories) vk.FreeMemory(device, memory, null); + vk.DestroyDevice(device, null); + Silk.NET.Core.Native.SilkMarshal.Free(extensions); + output.WriteLine($"pageable={pageable} priority={priority} type={deviceLocal}: wc +{(after.Bytes - before.Bytes) >> 20}MB committed, working set +{(workingSetAfter - workingSetBefore) >> 20}MB, for 512MB device-local"); + } +} diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/Float16ConversionDeviceTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/Float16ConversionDeviceTests.cs new file mode 100644 index 0000000..6d5fde0 --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/Float16ConversionDeviceTests.cs @@ -0,0 +1,181 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.Libs.Gpu.Buffers; +using SharpEmu.Libs.Tests.Gpu.Images; +using SharpEmu.ShaderCompiler; +using SharpEmu.ShaderCompiler.Resources; +using SharpEmu.ShaderCompiler.Tests; +using SharpEmu.ShaderCompiler.Vulkan; +using Xunit; +using static SharpEmu.ShaderCompiler.Tests.Resources.ResourceTestProgram; + +namespace SharpEmu.Libs.Tests.Gpu.Vulkan; + +// Packed f16 arithmetic with native f16<->f32 conversions (SupportsExactFloat16Conversions) +// against correctly rounded references, for every f16 value: two multiplies and a fused +// multiply-add, which widen each operand and round the result back to f16. +public sealed class Float16ConversionDeviceTests(HeadlessVulkanFixture fixture) : IClassFixture +{ + private const uint LocalSize = 1024; + + // (multiplier, addend) pairs, one f16 per lane: 1, 1/3 (rounds), the smallest denormal, + // the largest finite value, -0.75, and a NaN lane. + public static IEnumerable Operands() => + [ + [0x3C00_3C00u, 0x0000_8000u], + [0x3555_3555u, 0x0001_8001u], + [0x0001_8001u, 0x3C00_BC00u], + [0x7BFF_FBFFu, 0x7BFF_0400u], + [0xBA00_3A00u, 0x7E00_7C00u], + ]; + + [Theory] + [MemberData(nameof(Operands))] + public void PackedArithmeticIsCorrectlyRoundedForEveryHalf(uint multiplier, uint addend) + { + CheckPackedArithmetic(multiplier, addend, fmac: false); + CheckPackedArithmetic(multiplier, addend, fmac: true); + } + + private void CheckPackedArithmetic(uint multiplier, uint addend, bool fmac) + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) return; + if (!vulkan.ExactFloat16Conversions) + { + Assert.False(GatePrerequisites.DeviceRequired, "The required gate needs exact native float16 support."); + return; + } + var output = Run(vulkan, multiplier, addend, native: true, fmac); + var failures = new List(); + for (var value = 0; value < 0x10000; value++) + { + for (var result = 0; result < 3; result++) + { + uint expected = 0; + for (var lane = 0; lane < 2; lane++) + { + var x = (ushort)value; + var y = (ushort)(multiplier >> (16 * lane)); + var z = (ushort)(addend >> (16 * lane)); + var half = result switch + { + 0 => Multiply(x, y), + 1 => FusedMultiplyAdd(x, y, z), + _ => Multiply(x, x), + }; + expected |= (uint)half << (16 * lane); + } + + var actual = BitConverter.ToUInt32(output, value * 16 + result * 4); + if (!SameHalves(expected, actual) && failures.Count < 8) + { + failures.Add($"half=0x{value:X4} result={result} expected=0x{expected:X8} actual=0x{actual:X8}"); + } + } + } + + Assert.True(failures.Count == 0, string.Join(Environment.NewLine, failures)); + } + + // Equal halves, where any two NaNs are equal: the sign of a NaN result is not defined. + private static bool SameHalves(uint left, uint right) + { + for (var lane = 0; lane < 2; lane++) + { + var a = (ushort)(left >> (16 * lane)); + var b = (ushort)(right >> (16 * lane)); + var bothNan = (a & 0x7FFF) > 0x7C00 && (b & 0x7FFF) > 0x7C00; + if (a != b && !bothNan) return false; + } + + return true; + } + + private static double Widen(ushort bits) => (double)BitConverter.UInt16BitsToHalf(bits); + + private static ushort Narrow(double value) => BitConverter.HalfToUInt16Bits((Half)value); + + // A product of two halves is exact in double, so one rounding gives the f16 result. + private static ushort Multiply(ushort left, ushort right) => Narrow(Widen(left) * Widen(right)); + + // x * y + z rounded once to f16: the double sum is corrected to round-to-odd (2Sum gives + // the exact residual), which the round-to-nearest-even narrowing then rounds correctly. + private static ushort FusedMultiplyAdd(ushort x, ushort y, ushort z) + { + var product = Widen(x) * Widen(y); + var addend = Widen(z); + var sum = product + addend; + if (!double.IsFinite(sum)) return Narrow(sum); + var productPart = sum - addend; + var residual = (product - productPart) + (addend - (sum - productPart)); + var bits = BitConverter.DoubleToInt64Bits(sum); + if (residual != 0 && (bits & 1) == 0) + { + bits += Math.Sign(residual) == Math.Sign(sum) ? 1 : -1; + sum = BitConverter.Int64BitsToDouble(bits); + } + + return Narrow(sum); + } + + private static byte[] Run(HeadlessVulkan vulkan, uint multiplier, uint addend, bool native, bool fmac) + { + // v2 = s8 + v0 (the f16 value), v3 = that f16 in both lanes; + // v4 = v3 * s9, v5 = fma(v3, s9, s10), v6 = v3 * v3; stored at v2 * 16. + var fusedInstructions = fmac + ? new[] { MoveVectorFromScalar(24, 5, 10), Vop2(28, "VPkFmacF16", 5, Gen5Operand.Vector(3), Gen5Operand.Scalar(9)) } + : new[] { PackedF16(24, op: 0x0E, destination: 5, source0: 256 + 3, source1: 9, source2: 10) }; + var program = Program([ + MoveVectorFromScalar(0, 2, 8), + Vop2(4, "VAddU32", 2, Gen5Operand.Vector(0), Gen5Operand.Vector(2)), + Vop2(8, "VLshlrevB32", 3, Operand(16), Gen5Operand.Vector(2)), + Vop2(12, "VOrB32", 3, Gen5Operand.Vector(3), Gen5Operand.Vector(2)), + PackedF16(16, op: 0x10, destination: 4, source0: 256 + 3, source1: 9, source2: 0), + .. fusedInstructions, + PackedF16(32, op: 0x10, destination: 6, source0: 256 + 3, source1: 256 + 3, source2: 0), + Vop2(40, "VLshlrevB32", 1, Operand(4), Gen5Operand.Vector(2)), + BufferAccess(44, "BufferStoreDwordx3", 4, dwords: 3, vectorData: 4, offsetEnabled: true, vectorAddress: 1), + EndProgram(52)]); + var (plan, resources, layout) = Prepare(program); + var request = new ShaderCompileRequest(plan, resources, layout) + { + LocalSizeX = LocalSize, + SupportsExactFloat16Conversions = native, + }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + using var harness = new ImageTestHarness(vulkan); + using var runner = new LayoutComputeRunner(harness, request, shader.Spirv); + const uint bytes = 0x10000 * 16; + var output = runner.CreateBuffer(bytes); + var bindings = new Dictionary + { + [DescriptorBindingKind.Buffers] = Enumerable.Repeat(output, Math.Max(1, plan.Info.Buffers.Count)).ToArray(), + }; + var registers = new uint[256]; + registers[6] = bytes; + registers[9] = multiplier; + registers[10] = addend; + for (uint batch = 0; batch < 0x10000; batch += LocalSize) + { + registers[8] = batch; + harness.Run(() => runner.Dispatch(registers, bindings, 1)); + } + + harness.AssertNoValidationMessages(); + return runner.ReadBack(output, 0, bytes); + } + + // A VOP3P instruction with op_sel_hi set for every source (the high lane reads high halves). + private static Gen5ShaderInstruction PackedF16(uint pc, uint op, uint destination, uint source0, uint source1, uint source2) + { + var decoded = Gen5Float16ArithmeticTests.Decode( + [ + 0xCC00_0000u | (op << 16) | (1u << 14) | destination, + source0 | (source1 << 9) | (source2 << 18) | (3u << 27), + 0xBF81_0000u, + ]); + return decoded.Instructions[0] with { Pc = pc }; + } +} diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/FloatSemanticsDeviceTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/FloatSemanticsDeviceTests.cs index 9f7e37b..c2c66b9 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/FloatSemanticsDeviceTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/FloatSemanticsDeviceTests.cs @@ -122,5 +122,106 @@ public sealed class FloatSemanticsDeviceTests(HeadlessVulkanFixture fixture) : I } } + // op_sel_hi marks a V_FMA_MIX operand as f16, literals included: 0x34CD is 0.3, not an + // f32 denormal. Silent Hill weights a luminance with such literals and divides by it. + [Fact] + public void FusedMixReadsHalfSelectedLiteralsAsHalfFloats() + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) return; + var halfLiteral = new Gen5Vop3pControl(0, 0b001, 0, 0, false); + var floatLiteral = new Gen5Vop3pControl(0, 0b000, 0, 0, false); + var program = Program( + Vop1(0, "VCvtF32U32", 1, Gen5Operand.Vector(0)), + Vop3(4, "VFmaMixF32", 12, new Gen5Operand(Gen5OperandKind.LiteralConstant, 0x34CD), Gen5Operand.Vector(1), Operand(0)) with { Control = halfLiteral }, + Vop3(16, "VFmaMixF32", 13, new Gen5Operand(Gen5OperandKind.LiteralConstant, Bits(0.25f)), Gen5Operand.Vector(1), Operand(0)) with { Control = floatLiteral }, + Vop2(28, "VLshlrevB32", 7, Operand(5), Gen5Operand.Vector(0)), + BufferAccess(32, "BufferStoreDword", 4, vectorData: 12, offsetEnabled: true, vectorAddress: 7), + BufferAccess(40, "BufferStoreDword", 4, offset: 4, vectorData: 13, offsetEnabled: true, vectorAddress: 7), + EndProgram(48)); + var (plan, resources, layout) = Prepare(program); + var request = new ShaderCompileRequest(plan, resources, layout) + { + LocalSizeX = ThreadCount, ThreadCountX = ThreadCount, WaveSize = 32, + }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + using var harness = new ImageTestHarness(vulkan); + using var runner = new LayoutComputeRunner(harness, request, shader.Spirv); + var result = runner.CreateBuffer(ThreadCount * RowBytes); + var registers = new uint[256]; + registers[6] = ThreadCount * RowBytes; + harness.Run(() => runner.Dispatch(registers, + new Dictionary { [DescriptorBindingKind.Buffers] = [result] }, 1)); + var actual = runner.ReadBack(result, 0, ThreadCount * RowBytes); + var weight = (float)BitConverter.UInt16BitsToHalf(0x34CD); + for (var lane = 0u; lane < ThreadCount; lane++) + { + var row = actual.AsSpan((int)(lane * RowBytes)); + Assert.Equal(Bits(MathF.FusedMultiplyAdd(weight, lane, 0f)), BinaryPrimitives.ReadUInt32LittleEndian(row)); + Assert.Equal(Bits(lane * 0.25f), BinaryPrimitives.ReadUInt32LittleEndian(row[4..])); + } + } + + // V_CVT_PKRTZ_F16_F32 rounds toward zero: finite values past the f16 range saturate at + // +-65504 instead of becoming infinity, and subnormal results keep whole 2^-24 steps. + [Fact] + public void PackToHalfRoundsTowardZeroWithoutOverflowingToInfinity() + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) return; + var program = Program( + Vop1(0, "VCvtF32U32", 1, Gen5Operand.Vector(0)), + MoveScalar(4, 8, Bits(5003.7f)), + MoveScalar(12, 9, Bits(-80000.3f)), + MoveScalar(20, 10, Bits(3.3e-6f)), + MoveScalar(28, 11, Bits(-5e-5f)), + MoveScalar(36, 12, Bits(float.NaN)), + MoveScalar(44, 13, Bits(float.PositiveInfinity)), + Vop3(52, "VMadF32", 2, Gen5Operand.Vector(1), Gen5Operand.Scalar(8), Gen5Operand.Scalar(9)), + Vop3(60, "VMadF32", 3, Gen5Operand.Vector(1), Gen5Operand.Scalar(10), Gen5Operand.Scalar(11)), + Vop2(68, "VCvtPkrtzF16F32", 12, Gen5Operand.Vector(2), Gen5Operand.Vector(3)), + Vop3(72, "VCvtPkrtzF16F32", 13, Gen5Operand.Scalar(12), Gen5Operand.Scalar(13)), + Vop2(80, "VLshlrevB32", 7, Operand(5), Gen5Operand.Vector(0)), + BufferAccess(84, "BufferStoreDword", 4, vectorData: 12, offsetEnabled: true, vectorAddress: 7), + BufferAccess(92, "BufferStoreDword", 4, offset: 4, vectorData: 13, offsetEnabled: true, vectorAddress: 7), + EndProgram(100)); + var (plan, resources, layout) = Prepare(program); + var request = new ShaderCompileRequest(plan, resources, layout) + { + LocalSizeX = ThreadCount, ThreadCountX = ThreadCount, WaveSize = 32, + }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + using var harness = new ImageTestHarness(vulkan); + using var runner = new LayoutComputeRunner(harness, request, shader.Spirv); + var result = runner.CreateBuffer(ThreadCount * RowBytes); + var registers = new uint[256]; + registers[6] = ThreadCount * RowBytes; + harness.Run(() => runner.Dispatch(registers, + new Dictionary { [DescriptorBindingKind.Buffers] = [result] }, 1)); + var actual = runner.ReadBack(result, 0, ThreadCount * RowBytes); + for (var lane = 0u; lane < ThreadCount; lane++) + { + var row = actual.AsSpan((int)(lane * RowBytes)); + var large = MathF.FusedMultiplyAdd(lane, 5003.7f, -80000.3f); + var small = MathF.FusedMultiplyAdd(lane, 3.3e-6f, -5e-5f); + var packed = BinaryPrimitives.ReadUInt32LittleEndian(row); + Assert.Equal(HalfTowardZero(large), packed & 0xFFFF); + Assert.Equal(HalfTowardZero(small), packed >> 16); + var special = BinaryPrimitives.ReadUInt32LittleEndian(row[4..]); + Assert.True((special & 0x7C00) == 0x7C00 && (special & 0x3FF) != 0, $"lane {lane}: 0x{special:X8}"); + Assert.Equal(0x7C00u, special >> 16); + } + } + + private static uint HalfTowardZero(float value) + { + var sign = value < 0 || (value == 0 && float.IsNegative(value)) ? 0x8000u : 0u; + var magnitude = MathF.Abs(value); + if (magnitude >= 65504f) return sign | 0x7BFF; + if (magnitude < 6.10351562e-05f) return sign | (uint)Math.Floor(magnitude * 16777216.0); + var bits = BitConverter.SingleToUInt32Bits(magnitude); + return sign | ((((bits >> 23) - 127 + 15) << 10) | ((bits >> 13) & 0x3FF)); + } + private static uint Bits(float value) => BitConverter.SingleToUInt32Bits(value); } diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/GlobalMemoryShaderTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/GlobalMemoryShaderTests.cs index 0d6d16d..8b33bc6 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/GlobalMemoryShaderTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/GlobalMemoryShaderTests.cs @@ -160,6 +160,48 @@ public sealed class GlobalMemoryShaderTests(HeadlessVulkanFixture fixture, ITest Assert.Equal(expected, RunOnce(vulkan, shader, initial)); } + // A V# read from scalar-buffer data is used through its registers on the device. An 8-bit + // element in the last dword of a buffer whose size is not a multiple of four is in bounds: + // the hardware checks the element, not its dword (Ghost of Yotei's 19-byte state buffers). + [Theory] + [InlineData(14u, 19u, 5u)] + [InlineData(16u, 19u, 0xFFFF_FF81u)] + [InlineData(18u, 19u, 0xFFFF_FF81u)] + [InlineData(18u, 18u, 0u)] + public void DeviceDescriptorByteLoads_InTheLastPartialDwordReadMemory(uint offset, uint size, uint value) + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) return; + + var initial = CreateInput(); + initial.AsSpan(0, 20).Fill(0x81); + initial[14] = 5; + WriteWord(initial, DescriptorOffset, (uint)(BufferAddress & uint.MaxValue)); + WriteWord(initial, DescriptorOffset + 4, (uint)(BufferAddress >> 32)); + WriteWord(initial, DescriptorOffset + 8, size); + WriteWord(initial, DescriptorOffset + 12, (Format8Sint << 12) | IdentitySwizzle); + var expected = (byte[])initial.Clone(); + initial.AsSpan(0, 16).CopyTo(expected.AsSpan(MemoryOffset, 16)); + WriteWord(expected, MemoryOffset, value); + + var words = new List + { + 0xE0380000, 0x80000400, + // v_readfirstlane_b32 s20, v0: a lane-derived offset the plan cannot resolve. + 0x7E000000u | (20u << 17) | (2u << 9) | 256u, + // s_buffer_load_dwordx4 s[16:19], s[0:3], s20 offset:DescriptorOffset + 0xF4000000u | (10u << 18) | (16u << 6), (20u << 25) | DescriptorOffset, + 0xBF8CC07F, + 0xE0000000 | offset, 0x80040400, + 0xE0780000 | MemoryOffset, 0x80000400, + 0xBF810000, + }; + var shader = CompileProgram(DecodeProgram(words), BaseScalars()); + Assert.Equal(expected, RunOnce(vulkan, shader, initial)); + } + + private const uint Format8Sint = 6; + [Theory] [InlineData(1u, 188u, true)] [InlineData(2u, 184u, true)] diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/HeadlessVulkan.cs b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/HeadlessVulkan.cs index 5c3f36f..c4eb834 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/HeadlessVulkan.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/HeadlessVulkan.cs @@ -91,6 +91,13 @@ internal sealed unsafe class HeadlessVulkan : IDisposable public bool ShaderInt64 { get; } + // The runtime-descriptor GPU tests need an unbounded, dynamically indexed sampled-image + // array. Keep it opt-in: hosts without the descriptor-indexing feature simply skip them. + public bool SupportsRuntimeDescriptorArrays { get; private init; } + + // Created with shaderFloat16 and f16 float controls (VulkanFloat16Support). + public bool ExactFloat16Conversions { get; private init; } + public bool ValidationEnabled => _debugUtils is not null; // The validation messages collected since the last call; empty when the layer is off. @@ -128,8 +135,9 @@ internal sealed unsafe class HeadlessVulkan : IDisposable { return Create(); } - catch (Exception) + catch (Exception exception) { + Console.Error.WriteLine($"[TEST][VULKAN] Device setup failed: {exception}"); return null; } } @@ -161,6 +169,7 @@ internal sealed unsafe class HeadlessVulkan : IDisposable public void Dispose() { Vk.DeviceWaitIdle(Device); + _deviceInfo?.Slabs.Destroy(); Vk.DestroyDevice(Device, null); if (_debugUtils is { } debugUtils) { @@ -168,6 +177,7 @@ internal sealed unsafe class HeadlessVulkan : IDisposable } Vk.DestroyInstance(Instance, null); + AssertNoValidationMessages(); } private static uint DebugCallback(DebugUtilsMessageSeverityFlagsEXT severity, DebugUtilsMessageTypeFlagsEXT type, DebugUtilsMessengerCallbackDataEXT* data, void* userData) @@ -215,22 +225,33 @@ internal sealed unsafe class HeadlessVulkan : IDisposable var appInfo = new ApplicationInfo { SType = StructureType.ApplicationInfo, ApiVersion = apiVersion }; var validation = Environment.GetEnvironmentVariable(ValidationVariable) == "1"; var layers = validation ? SilkMarshal.StringArrayToPtr(new[] { "VK_LAYER_KHRONOS_validation" }) : 0; - var extensions = validation ? SilkMarshal.StringArrayToPtr(new[] { ExtDebugUtils.ExtensionName }) : 0; + uint instanceExtensionCount = 0; + vk.EnumerateInstanceExtensionProperties((byte*)null, &instanceExtensionCount, null); + var instanceExtensions = new ExtensionProperties[instanceExtensionCount]; + fixed (ExtensionProperties* pointer = instanceExtensions) + vk.EnumerateInstanceExtensionProperties((byte*)null, &instanceExtensionCount, pointer); + var portable = instanceExtensions.Any(extension => + SilkMarshal.PtrToString((nint)extension.ExtensionName) == "VK_KHR_portability_enumeration"); + var instanceNames = new List(); + if (validation) instanceNames.Add(ExtDebugUtils.ExtensionName); + if (portable) instanceNames.Add("VK_KHR_portability_enumeration"); + var extensions = instanceNames.Count != 0 ? SilkMarshal.StringArrayToPtr(instanceNames.ToArray()) : 0; var instanceInfo = new InstanceCreateInfo { SType = StructureType.InstanceCreateInfo, + Flags = portable ? InstanceCreateFlags.EnumeratePortabilityBitKhr : 0, PApplicationInfo = &appInfo, EnabledLayerCount = validation ? 1u : 0u, PpEnabledLayerNames = (byte**)layers, - EnabledExtensionCount = validation ? 1u : 0u, + EnabledExtensionCount = (uint)instanceNames.Count, PpEnabledExtensionNames = (byte**)extensions, }; var created = vk.CreateInstance(&instanceInfo, null, out var instance); if (validation) { SilkMarshal.Free(layers); - SilkMarshal.Free(extensions); } + if (extensions != 0) SilkMarshal.Free(extensions); if (created != Result.Success) { @@ -318,10 +339,21 @@ internal sealed unsafe class HeadlessVulkan : IDisposable SType = StructureType.PhysicalDeviceTimelineSemaphoreFeatures, PNext = &addressFeatures, }; - var features = new PhysicalDeviceFeatures2 { SType = StructureType.PhysicalDeviceFeatures2, PNext = &timelineFeatures }; + var descriptorIndexingFeatures = new PhysicalDeviceDescriptorIndexingFeatures + { + SType = StructureType.PhysicalDeviceDescriptorIndexingFeatures, + PNext = &timelineFeatures, + }; + var features = new PhysicalDeviceFeatures2 { SType = StructureType.PhysicalDeviceFeatures2, PNext = &descriptorIndexingFeatures }; vk.GetPhysicalDeviceFeatures2(physical, &features); var dynamicRendering = vulkan13Features.DynamicRendering && vulkan13Features.Synchronization2 && HasDeviceExtensions(vk, physical, RenderingExtensionNames); + var runtimeDescriptorArrays = descriptorIndexingFeatures.ShaderSampledImageArrayNonUniformIndexing && + descriptorIndexingFeatures.RuntimeDescriptorArray && + descriptorIndexingFeatures.DescriptorBindingPartiallyBound && + descriptorIndexingFeatures.DescriptorBindingSampledImageUpdateAfterBind && + descriptorIndexingFeatures.DescriptorBindingUpdateUnusedWhilePending && + descriptorIndexingFeatures.DescriptorBindingVariableDescriptorCount; if (family == uint.MaxValue || !timelineFeatures.TimelineSemaphore || !addressFeatures.BufferDeviceAddress || !vulkan13Features.Synchronization2) { @@ -332,6 +364,7 @@ internal sealed unsafe class HeadlessVulkan : IDisposable vk.GetPhysicalDeviceFeatures(physical, out var baseFeatures); var enabledFeatures = new PhysicalDeviceFeatures { + TessellationShader = baseFeatures.TessellationShader, SampleRateShading = baseFeatures.SampleRateShading, SamplerAnisotropy = baseFeatures.SamplerAnisotropy, ShaderStorageImageExtendedFormats = baseFeatures.ShaderStorageImageExtendedFormats, @@ -362,12 +395,36 @@ internal sealed unsafe class HeadlessVulkan : IDisposable PNext = &vulkan13Features, }; timelineFeatures.PNext = &addressFeatures; + descriptorIndexingFeatures = new PhysicalDeviceDescriptorIndexingFeatures + { + SType = StructureType.PhysicalDeviceDescriptorIndexingFeatures, + ShaderSampledImageArrayNonUniformIndexing = runtimeDescriptorArrays, + RuntimeDescriptorArray = runtimeDescriptorArrays, + DescriptorBindingPartiallyBound = runtimeDescriptorArrays, + DescriptorBindingSampledImageUpdateAfterBind = runtimeDescriptorArrays, + DescriptorBindingUpdateUnusedWhilePending = runtimeDescriptorArrays, + DescriptorBindingVariableDescriptorCount = runtimeDescriptorArrays, + PNext = &timelineFeatures, + }; if (barycentric) { barycentricFeatures.PNext = vulkan13Features.PNext; vulkan13Features.PNext = &barycentricFeatures; } + + var exactFloat16 = SharpEmu.Libs.VideoOut.VulkanFloat16Support.SupportsExactConversions(vk, physical); + var float16Features = new PhysicalDeviceShaderFloat16Int8Features + { + SType = StructureType.PhysicalDeviceShaderFloat16Int8Features, + ShaderFloat16 = true, + }; + if (exactFloat16) + { + float16Features.PNext = vulkan13Features.PNext; + vulkan13Features.PNext = &float16Features; + } var extensionNames = new List(); + if (HasDeviceExtensions(vk, physical, ["VK_KHR_portability_subset"])) extensionNames.Add("VK_KHR_portability_subset"); if (dynamicRendering) extensionNames.AddRange(RenderingExtensionNames); if (barycentric) extensionNames.Add(barycentricExtension); const string fillRectangleExtension = "VK_NV_fill_rectangle"; @@ -377,7 +434,7 @@ internal sealed unsafe class HeadlessVulkan : IDisposable var deviceInfo = new DeviceCreateInfo { SType = StructureType.DeviceCreateInfo, - PNext = &timelineFeatures, + PNext = &descriptorIndexingFeatures, QueueCreateInfoCount = 1, PQueueCreateInfos = &queueInfo, PEnabledFeatures = &enabledFeatures, @@ -401,6 +458,8 @@ internal sealed unsafe class HeadlessVulkan : IDisposable { SupportsFragmentShaderBarycentric = barycentric, SupportsFillRectangle = fillRectangle, + ExactFloat16Conversions = exactFloat16, + SupportsRuntimeDescriptorArrays = runtimeDescriptorArrays, }; if (validation) { diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/LayoutComputeRunner.cs b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/LayoutComputeRunner.cs index 708cba0..97201ed 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/LayoutComputeRunner.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/LayoutComputeRunner.cs @@ -19,6 +19,7 @@ internal sealed unsafe class LayoutComputeRunner : IDisposable { private readonly ImageTestHarness _harness; private readonly ShaderCompileRequest _request; + private readonly DescriptorSetLayout _globalSetLayout; private readonly DescriptorSetLayout _setLayout; private readonly PipelineLayout _pipelineLayout; private readonly DescriptorPool _pool; @@ -33,10 +34,12 @@ internal sealed unsafe class LayoutComputeRunner : IDisposable : ImageDescriptorBinding.ResourceClass(kind) == ImageResourceClass.Storage ? DescriptorType.StorageImage : DescriptorType.StorageBuffer; - public LayoutComputeRunner(ImageTestHarness harness, ShaderCompileRequest request, byte[] spirv) + public LayoutComputeRunner(ImageTestHarness harness, ShaderCompileRequest request, byte[] spirv, + DescriptorSetLayout globalSetLayout = default) { _harness = harness; _request = request; + _globalSetLayout = globalSetLayout; var vk = harness.Vk; var device = harness.Device.Device; var descriptors = request.Bindings.Descriptors; @@ -59,13 +62,27 @@ internal sealed unsafe class LayoutComputeRunner : IDisposable }; Require(vk.CreateDescriptorSetLayout(device, &layoutInfo, null, out _setLayout), "vkCreateDescriptorSetLayout"); var push = new PushConstantRange(ShaderStageFlags.ComputeBit, 0, PushData.ByteSize); - fixed (DescriptorSetLayout* layout = &_setLayout) + if (_globalSetLayout.Handle != 0) { + var layouts = stackalloc DescriptorSetLayout[] { _globalSetLayout, _setLayout }; + var pipelineLayoutInfo = new PipelineLayoutCreateInfo + { + SType = StructureType.PipelineLayoutCreateInfo, + SetLayoutCount = 2, + PSetLayouts = layouts, + PushConstantRangeCount = request.Bindings.UsesPushData ? 1u : 0u, + PPushConstantRanges = &push, + }; + Require(vk.CreatePipelineLayout(device, &pipelineLayoutInfo, null, out _pipelineLayout), "vkCreatePipelineLayout"); + } + else + { + var layout = _setLayout; var pipelineLayoutInfo = new PipelineLayoutCreateInfo { SType = StructureType.PipelineLayoutCreateInfo, SetLayoutCount = 1, - PSetLayouts = layout, + PSetLayouts = &layout, PushConstantRangeCount = request.Bindings.UsesPushData ? 1u : 0u, PPushConstantRanges = &push, }; @@ -181,12 +198,21 @@ internal sealed unsafe class LayoutComputeRunner : IDisposable uint[]? flattenedTable = null, IReadOnlyDictionary? boundImages = null, ulong shaderBase = 0, - uint[]? dispatchThreadLimits = null) + uint[]? dispatchThreadLimits = null, + uint[]? bufferStrides = null, + uint[]? tessellationData = null, + DescriptorSet globalSet = default) { var vk = _harness.Vk; var device = _harness.Device.Device; var layout = _request.Bindings; var shaderData = new uint[layout.ShaderDataDwordCount]; + if (layout.UsesTessellationData) + { + if (tessellationData is not { Length: (int)Gen5TessellationData.DwordCount }) + throw new InvalidOperationException("The tessellation kernel requires its runtime data."); + tessellationData.CopyTo(shaderData, (int)layout.TessellationDataDword); + } for (var index = 0; index < layout.UserDataRegisters.Count; index++) { var register = layout.UserDataRegisters[index]; @@ -199,6 +225,15 @@ internal sealed unsafe class LayoutComputeRunner : IDisposable shaderData[layout.ShaderBaseDword + 1] = (uint)(shaderBase >> 32); } + if (layout.UsesRuntimeBufferStrides) + { + // The host packs each buffer's descriptor stride, two per dword. + for (var index = 0; index < (bufferStrides?.Length ?? 0); index++) + { + shaderData[layout.BufferStrideDword + (uint)index / 2] |= (bufferStrides![index] & 0x3FFF) << ((index % 2) * 16); + } + } + if (layout.UsesDispatchThreadLimits) { if (dispatchThreadLimits is not { Length: 3 }) throw new InvalidOperationException("The dispatch requires three thread limits."); @@ -297,7 +332,20 @@ internal sealed unsafe class LayoutComputeRunner : IDisposable }; VulkanSynchronization.PipelineBarrier(vk, command, PipelineStageFlags.AllCommandsBit | PipelineStageFlags.HostBit, PipelineStageFlags.ComputeShaderBit, 0, 1, &barrier, 0, null, 0, null); vk.CmdBindPipeline(command, PipelineBindPoint.Compute, _pipeline); - vk.CmdBindDescriptorSets(command, PipelineBindPoint.Compute, _pipelineLayout, 0, 1, &set, 0, null); + if (_globalSetLayout.Handle != 0) + { + if (globalSet.Handle == 0) + { + throw new InvalidOperationException("This pipeline needs the bindless descriptor set."); + } + + vk.CmdBindDescriptorSets(command, PipelineBindPoint.Compute, _pipelineLayout, 0, 1, &globalSet, 0, null); + vk.CmdBindDescriptorSets(command, PipelineBindPoint.Compute, _pipelineLayout, 1, 1, &set, 0, null); + } + else + { + vk.CmdBindDescriptorSets(command, PipelineBindPoint.Compute, _pipelineLayout, 0, 1, &set, 0, null); + } if (layout.UsesPushData) { fixed (uint* pointer = pushData) @@ -341,3 +389,105 @@ internal sealed unsafe class LayoutComputeRunner : IDisposable vk.DestroyDescriptorSetLayout(device, _setLayout, null); } } + +// Tiny bindless set for runtime-descriptor GPU coverage. The production heap is persistent and +// much larger; this deliberately keeps two fully populated sampled-image slots: zero is the null +// image and one is the host-registered descriptor. +internal sealed unsafe class RuntimeDescriptorImageHeap : IDisposable +{ + private readonly ImageTestHarness _harness; + private readonly DescriptorPool _pool; + + public RuntimeDescriptorImageHeap(ImageTestHarness harness) + { + _harness = harness; + var vk = harness.Vk; + var device = harness.Device.Device; + var binding = new DescriptorSetLayoutBinding( + BindingLayout.BindlessSampledImageBinding, + DescriptorType.SampledImage, + 2, + ShaderStageFlags.ComputeBit); + var flags = DescriptorBindingFlags.PartiallyBoundBit | DescriptorBindingFlags.UpdateAfterBindBit | + DescriptorBindingFlags.UpdateUnusedWhilePendingBit | DescriptorBindingFlags.VariableDescriptorCountBit; + var bindingFlags = new DescriptorSetLayoutBindingFlagsCreateInfo + { + SType = StructureType.DescriptorSetLayoutBindingFlagsCreateInfo, + BindingCount = 1, + PBindingFlags = &flags, + }; + var layoutInfo = new DescriptorSetLayoutCreateInfo + { + SType = StructureType.DescriptorSetLayoutCreateInfo, + PNext = &bindingFlags, + Flags = DescriptorSetLayoutCreateFlags.UpdateAfterBindPoolBit, + BindingCount = 1, + PBindings = &binding, + }; + Require(vk.CreateDescriptorSetLayout(device, &layoutInfo, null, out var createdLayout), "vkCreateDescriptorSetLayout(runtime heap)"); + Layout = createdLayout; + + var poolSize = new DescriptorPoolSize(DescriptorType.SampledImage, 2); + var poolInfo = new DescriptorPoolCreateInfo + { + SType = StructureType.DescriptorPoolCreateInfo, + Flags = DescriptorPoolCreateFlags.UpdateAfterBindBit, + MaxSets = 1, + PoolSizeCount = 1, + PPoolSizes = &poolSize, + }; + Require(vk.CreateDescriptorPool(device, &poolInfo, null, out _pool), "vkCreateDescriptorPool(runtime heap)"); + var descriptorCount = 2u; + var variableCounts = new DescriptorSetVariableDescriptorCountAllocateInfo + { + SType = StructureType.DescriptorSetVariableDescriptorCountAllocateInfo, + DescriptorSetCount = 1, + PDescriptorCounts = &descriptorCount, + }; + var layout = Layout; + var allocate = new DescriptorSetAllocateInfo + { + SType = StructureType.DescriptorSetAllocateInfo, + PNext = &variableCounts, + DescriptorPool = _pool, + DescriptorSetCount = 1, + PSetLayouts = &layout, + }; + Require(vk.AllocateDescriptorSets(device, &allocate, out var allocatedSet), "vkAllocateDescriptorSets(runtime heap)"); + Set = allocatedSet; + } + + public DescriptorSetLayout Layout { get; } + + public DescriptorSet Set { get; } + + public void Write(uint slot, ImageView view) + { + var info = new DescriptorImageInfo { ImageView = view, ImageLayout = ImageLayout.ShaderReadOnlyOptimal }; + var write = new WriteDescriptorSet + { + SType = StructureType.WriteDescriptorSet, + DstSet = Set, + DstBinding = BindingLayout.BindlessSampledImageBinding, + DstArrayElement = slot, + DescriptorCount = 1, + DescriptorType = DescriptorType.SampledImage, + PImageInfo = &info, + }; + _harness.Vk.UpdateDescriptorSets(_harness.Device.Device, 1, &write, 0, null); + } + + public void Dispose() + { + _harness.Vk.DestroyDescriptorPool(_harness.Device.Device, _pool, null); + _harness.Vk.DestroyDescriptorSetLayout(_harness.Device.Device, Layout, null); + } + + private static void Require(Result result, string operation) + { + if (result != Result.Success) + { + throw new InvalidOperationException($"{operation} failed with {result}"); + } + } +} diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/LayoutImageShaderTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/LayoutImageShaderTests.cs index cfd43c6..97f4e0e 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/LayoutImageShaderTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/LayoutImageShaderTests.cs @@ -20,7 +20,7 @@ namespace SharpEmu.Libs.Tests.Gpu.Vulkan; // Images and samplers bound through the class arrays of a compile request: element // selection, the duplicated point sampler, and the per-mip storage descriptors. -public sealed class LayoutImageShaderTests(HeadlessVulkanFixture fixture, ITestOutputHelper output) : IClassFixture +public sealed class LayoutImageShaderTests(HeadlessVulkanFixture fixture, ITestOutputHelper? output) : IClassFixture { private const uint Format32Uint = 20; private const uint Format32Sint = 21; @@ -92,6 +92,17 @@ public sealed class LayoutImageShaderTests(HeadlessVulkanFixture fixture, ITestO return Program([.. instructions]); } + // The descriptor's first dword comes from a lane. It is zero for this one-lane dispatch and + // is valid at execution, but has no plan-time scalar source for the resource tracker. + // With inactiveLane the load runs with EXEC cleared, as a lane whose result is unused. + private static Gen5ShaderProgram RuntimeImageLoadProgram(bool inactiveLane = false) => Program( + ReadFirstLane(0, FirstImageRegister, 1), + MoveVector(8, 0, 0), + MoveScalar(12, inactiveLane ? 126u : 100u, 0), + Image(16, "ImageLoad", FirstImageRegister, vectorAddress: 0, dmask: 1), + BufferAccess(24, "BufferStoreDword", ResultRegister, 0, 1, vectorData: 4), + EndProgram(32)); + private static ImageDescription Describe(ulong guestOffset, Format format, GuestPixelFormat guestFormat, uint width, uint height, uint levels) { var description = ImageDescription.Create(); @@ -221,6 +232,44 @@ public sealed class LayoutImageShaderTests(HeadlessVulkanFixture fixture, ITestO ImageLayout = ImageLayout.General, }; + [Theory] + [InlineData(-32768)] + [InlineData(-1)] + [InlineData(32767)] + public void Signed16StorageImage_PreservesSignedLoadsAndStores(short input) + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan)) return; + var instructions = new List(); + uint pc = 0; + instructions.AddRange(ImageWords(ref pc, FirstImageRegister, FirstImageAddress, 12)); + instructions.Add(MoveVector(pc, 1, 0)); pc += 8; + instructions.Add(MoveVector(pc, 2, 0)); pc += 8; + instructions.Add(Image(pc, "ImageLoad", FirstImageRegister, vectorAddress: 1, dmask: 1)); pc += 8; + instructions.Add(BufferAccess(pc, "BufferStoreDword", ResultRegister, 0, 1, vectorData: 4)); pc += 8; + instructions.Add(MoveVector(pc, 4, unchecked((uint)-12345))); pc += 8; + instructions.Add(Image(pc, "ImageStore", FirstImageRegister, vectorAddress: 1, dmask: 1)); pc += 8; + instructions.Add(EndProgram(pc)); + using var run = new Run(vulkan, Program([.. instructions]), UserData(), 1); + Assert.All(run.Resources.Info.Images, resource => Assert.Equal(ImageNumericClass.Sint, resource.NumericClass)); + var image = run.Harness.CreateImage(Describe(0, Format.R16Sint, GuestPixelFormat.Bits16SInt, 4, 4, 1)); + var pixels = Enumerable.Repeat(input, 16).ToArray(); + var copies = ImageTestHarness.WholeImageCopies(image.Description, 0); + run.Harness.UploadImage(image, MemoryMarshal.AsBytes(pixels.AsSpan()), copies); + var view = new DescriptorImageInfo + { + ImageView = image.GetOrCreateView(ImageViewDescription.Default with { Format = Format.R16Sint, Usage = ImageUsageFlags.SampledBit | ImageUsageFlags.StorageBit, LevelCount = 1 }), + ImageLayout = ImageLayout.General, + }; + var views = Enumerable.Range(0, run.Snapshot.Images.Length).ToDictionary(index => index, _ => new[] { view }); + run.Dispatch(run.BindImages(views), command => image.Transition(ImageLayout.General, AccessFlags.ShaderReadBit | AccessFlags.ShaderWriteBit, null, command)); + Assert.Equal(unchecked((uint)(int)input), run.ResultWord(0)); + var actual = MemoryMarshal.Cast(run.Harness.ReadImage(image, copies, 32)).ToArray(); + pixels[0] = -12345; + Assert.Equal(pixels, actual); + run.Harness.AssertNoValidationMessages(); + } + [Fact] public void SampledClassArray_SamplesEachElementThroughTheLayout() { @@ -257,6 +306,139 @@ public sealed class LayoutImageShaderTests(HeadlessVulkanFixture fixture, ITestO output.WriteLine($"Verified sampled class arrays on {vulkan.DeviceName}; validation={vulkan.ValidationEnabled}."); } + [Fact] + public void RuntimeImageLoad_UsesTheRegisteredUnsignedHeapViewAfterItsMiss() + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan) || !vulkan!.SupportsRuntimeDescriptorArrays) + { + return; + } + + uint[] words = [0, Format32Uint << 20, 3 | (3 << 14), 0xFAC | (ImageType2D << 28), 0, 0, 0, 0]; + var program = RuntimeImageLoadProgram(); + var plan = ShaderResourcePlan.Extract(program, ShaderStage.Compute, 0x5255_4E54_494D_47, 0, 64); + Assert.True(plan.Info.UsesRuntimeDescriptors); + Assert.Empty(plan.Info.NullDescriptorFallbacks); + var resources = ResourceMaterializer.ApplyTo(plan, ResourceSpecialization.Default(plan.Info)); + var layout = BindingLayout.Allocate(resources.Info, + BindingLayout.CollectUserDataRegisters(program, 0, 64), false, + ShaderCompileRequest.RequiresFlattenedTable(plan, resources), false, + usesBindlessImages: true); + var request = new ShaderCompileRequest(plan, resources, layout) { ThreadCountX = 1, ThreadCountY = 1, ThreadCountZ = 1 }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + + using var harness = new ImageTestHarness(vulkan); + using var heap = new RuntimeDescriptorImageHeap(harness); + var zero = harness.CreateImage(Describe(0, Format.R32Uint, GuestPixelFormat.Bits32UInt, 1, 1, 1)); + var value = harness.CreateImage(Describe(0x10000, Format.R32Uint, GuestPixelFormat.Bits32UInt, 1, 1, 1)); + harness.UploadImage(zero, MemoryMarshal.AsBytes([0u]), ImageTestHarness.WholeImageCopies(zero.Description, 0)); + harness.UploadImage(value, MemoryMarshal.AsBytes([0xCAFE_BABEu]), ImageTestHarness.WholeImageCopies(value.Description, 0)); + var zeroView = SampledView(zero).ImageView; + var valueView = SampledView(value).ImageView; + heap.Write(0, zeroView); + heap.Write(1, valueView); + using var runner = new LayoutComputeRunner(harness, request, shader.Spirv, heap.Layout); + var result = runner.CreateBuffer(ResultBytes); + var registers = new uint[256]; + words[1..].CopyTo(registers, (int)FirstImageRegister + 1); + registers[ResultRegister + 2] = ResultBytes; + var key = new uint[1 + words.Length]; + key[0] = RuntimeDescriptorTable.ViewClass(ImageDimension.Dim2D, ImageNumericClass.Uint); + words.CopyTo(key, 1); + + var firstMisses = runner.CreateBuffer((ulong)RuntimeDescriptorTable.MissBufferDwords * sizeof(uint)); + harness.Run(() => + { + var command = new CommandBuffer(harness.Scheduler.Current.Handle); + zero.Transition(ImageLayout.ShaderReadOnlyOptimal, AccessFlags.ShaderReadBit, null, command); + value.Transition(ImageLayout.ShaderReadOnlyOptimal, AccessFlags.ShaderReadBit, null, command); + runner.Dispatch(registers, new Dictionary + { + [DescriptorBindingKind.Buffers] = [result], + [DescriptorBindingKind.RuntimeDescriptorTable] = [runner.CreateBuffer(MemoryMarshal.AsBytes(RuntimeDescriptorTable.Build([], [])))], + [DescriptorBindingKind.RuntimeDescriptorMisses] = [firstMisses], + }, 1, globalSet: heap.Set); + }); + Assert.Equal(0u, BinaryPrimitives.ReadUInt32LittleEndian(runner.ReadBack(result, 0, 4))); + var firstMissWords = MemoryMarshal.Cast(runner.ReadBack(firstMisses, 0, (ulong)RuntimeDescriptorTable.MissBufferDwords * sizeof(uint))); + Assert.Equal(1u, firstMissWords[(int)RuntimeDescriptorTable.ImageMissCountDword]); + Assert.Equal(key, firstMissWords.Slice((int)RuntimeDescriptorTable.ImageMissOffset, key.Length).ToArray()); + + // The miss is what causes the host to insert the already-created typed R32_UINT view + // for the next dispatch. Slot zero remains the null texture for the first dispatch. + harness.Finish(); + var secondMisses = runner.CreateBuffer((ulong)RuntimeDescriptorTable.MissBufferDwords * sizeof(uint)); + harness.Run(() => + { + var command = new CommandBuffer(harness.Scheduler.Current.Handle); + zero.Transition(ImageLayout.ShaderReadOnlyOptimal, AccessFlags.ShaderReadBit, null, command); + value.Transition(ImageLayout.ShaderReadOnlyOptimal, AccessFlags.ShaderReadBit, null, command); + runner.Dispatch(registers, new Dictionary + { + [DescriptorBindingKind.Buffers] = [result], + [DescriptorBindingKind.RuntimeDescriptorTable] = [runner.CreateBuffer(MemoryMarshal.AsBytes(RuntimeDescriptorTable.Build([(key, 1)], [])))], + [DescriptorBindingKind.RuntimeDescriptorMisses] = [secondMisses], + }, 1, globalSet: heap.Set); + }); + + var secondMissWords = MemoryMarshal.Cast(runner.ReadBack(secondMisses, 0, (ulong)RuntimeDescriptorTable.MissBufferDwords * sizeof(uint))); + Assert.Equal(0u, secondMissWords[(int)RuntimeDescriptorTable.ImageMissCountDword]); + Assert.Equal(0xCAFE_BABEu, BinaryPrimitives.ReadUInt32LittleEndian(runner.ReadBack(result, 0, 4))); + harness.AssertNoValidationMessages(); + output?.WriteLine($"Verified runtime R32_UINT ImageLoad after descriptor registration on {vulkan.DeviceName}."); + } + + // A lane EXEC disables does not access its image, so its descriptor is not a miss. + [Fact] + public void RuntimeImageLoad_InactiveLaneRecordsNoMiss() + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan) || !vulkan!.SupportsRuntimeDescriptorArrays) + { + return; + } + + uint[] words = [0, Format32Uint << 20, 3 | (3 << 14), 0xFAC | (ImageType2D << 28), 0, 0, 0, 0]; + var program = RuntimeImageLoadProgram(inactiveLane: true); + var plan = ShaderResourcePlan.Extract(program, ShaderStage.Compute, 0x5255_4E54_494D_48, 0, 64); + Assert.True(plan.Info.UsesRuntimeDescriptors); + var resources = ResourceMaterializer.ApplyTo(plan, ResourceSpecialization.Default(plan.Info)); + var layout = BindingLayout.Allocate(resources.Info, + BindingLayout.CollectUserDataRegisters(program, 0, 64), false, + ShaderCompileRequest.RequiresFlattenedTable(plan, resources), false, + usesBindlessImages: true); + var request = new ShaderCompileRequest(plan, resources, layout) { ThreadCountX = 1, ThreadCountY = 1, ThreadCountZ = 1 }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + + using var harness = new ImageTestHarness(vulkan); + using var heap = new RuntimeDescriptorImageHeap(harness); + var zero = harness.CreateImage(Describe(0, Format.R32Uint, GuestPixelFormat.Bits32UInt, 1, 1, 1)); + harness.UploadImage(zero, MemoryMarshal.AsBytes([0u]), ImageTestHarness.WholeImageCopies(zero.Description, 0)); + heap.Write(0, SampledView(zero).ImageView); + heap.Write(1, SampledView(zero).ImageView); + using var runner = new LayoutComputeRunner(harness, request, shader.Spirv, heap.Layout); + var registers = new uint[256]; + words[1..].CopyTo(registers, (int)FirstImageRegister + 1); + registers[ResultRegister + 2] = ResultBytes; + var misses = runner.CreateBuffer((ulong)RuntimeDescriptorTable.MissBufferDwords * sizeof(uint)); + harness.Run(() => + { + var command = new CommandBuffer(harness.Scheduler.Current.Handle); + zero.Transition(ImageLayout.ShaderReadOnlyOptimal, AccessFlags.ShaderReadBit, null, command); + runner.Dispatch(registers, new Dictionary + { + [DescriptorBindingKind.Buffers] = [runner.CreateBuffer(ResultBytes)], + [DescriptorBindingKind.RuntimeDescriptorTable] = [runner.CreateBuffer(MemoryMarshal.AsBytes(RuntimeDescriptorTable.Build([], [])))], + [DescriptorBindingKind.RuntimeDescriptorMisses] = [misses], + }, 1, globalSet: heap.Set); + }); + + var missWords = MemoryMarshal.Cast(runner.ReadBack(misses, 0, (ulong)RuntimeDescriptorTable.MissBufferDwords * sizeof(uint))); + Assert.Equal(0u, missWords[(int)RuntimeDescriptorTable.ImageMissCountDword]); + harness.AssertNoValidationMessages(); + } + [Fact] public void PointOnlyImage_SamplesThroughTheDuplicatedSampler() { diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/LdsOutOfRangeDeviceTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/LdsOutOfRangeDeviceTests.cs new file mode 100644 index 0000000..de40090 --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/LdsOutOfRangeDeviceTests.cs @@ -0,0 +1,62 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.Libs.Gpu.Buffers; +using SharpEmu.Libs.Tests.Gpu.Images; +using SharpEmu.ShaderCompiler; +using SharpEmu.ShaderCompiler.Resources; +using SharpEmu.ShaderCompiler.Vulkan; +using Xunit; +using static SharpEmu.ShaderCompiler.Tests.Resources.ResourceTestProgram; + +namespace SharpEmu.Libs.Tests.Gpu.Vulkan; + +// An LDS access past the allocation reads 0 and drops writes. Ghost of Yotei's exposure blur +// (0xD06A706D2994D2B1) reads neighbours at negative addresses for its edge lanes; wrapping them +// into the array read the wave64 scratch instead and turned the exposure into NaN. +public sealed class LdsOutOfRangeDeviceTests(HeadlessVulkanFixture fixture) : IClassFixture +{ + [Fact] + public void OutOfRangeReadsZeroAndWritesAreDropped() + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) return; + const uint Pattern = 0x7FC0_1234; + var program = Program( + Vop2(0, "VLshlrevB32", 1, Operand(2), Gen5Operand.Vector(0)), + MoveVectorFromScalar(4, 2, 8), + MoveVectorFromScalar(8, 3, 9), + // LDS[lane] = pattern; a write at 0xFFFFFFF0 must be dropped. + DataShare(12, "DsWriteB32", false, [Gen5Operand.Vector(1), Gen5Operand.Vector(2)], []), + DataShare(20, "DsWriteB32", false, [Gen5Operand.Vector(3), Gen5Operand.Vector(2)], []), + // v4 = LDS[0xFFFFFFF0] (out of range: 0), v5 = LDS[lane] (the pattern). + DataShare(28, "DsReadB32", false, [Gen5Operand.Vector(3)], [4]), + DataShare(36, "DsReadB32", false, [Gen5Operand.Vector(1)], [5]), + Vop2(44, "VLshlrevB32", 6, Operand(3), Gen5Operand.Vector(0)), + BufferAccess(48, "BufferStoreDwordx2", 4, dwords: 2, vectorData: 4, offsetEnabled: true, vectorAddress: 6), + EndProgram(56)); + var (plan, resources, layout) = Prepare(program); + var request = new ShaderCompileRequest(plan, resources, layout) { LocalSizeX = 64 }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + using var harness = new ImageTestHarness(vulkan); + using var runner = new LayoutComputeRunner(harness, request, shader.Spirv); + const uint bytes = 64 * 8; + var output = runner.CreateBuffer(bytes); + var bindings = new Dictionary + { + [DescriptorBindingKind.Buffers] = Enumerable.Repeat(output, Math.Max(1, plan.Info.Buffers.Count)).ToArray(), + }; + var registers = new uint[256]; + registers[6] = bytes; + registers[8] = Pattern; + registers[9] = 0xFFFF_FFF0; + harness.Run(() => runner.Dispatch(registers, bindings, 1)); + harness.AssertNoValidationMessages(); + var data = runner.ReadBack(output, 0, bytes); + for (var lane = 0; lane < 64; lane++) + { + Assert.Equal(0u, BitConverter.ToUInt32(data, lane * 8)); + Assert.Equal(Pattern, BitConverter.ToUInt32(data, lane * 8 + 4)); + } + } +} diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/MoveRelativeRegisterDeviceTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/MoveRelativeRegisterDeviceTests.cs new file mode 100644 index 0000000..c8af168 --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/MoveRelativeRegisterDeviceTests.cs @@ -0,0 +1,46 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Buffers.Binary; +using SharpEmu.Libs.Gpu.Buffers; +using SharpEmu.Libs.Tests.Gpu.Images; +using SharpEmu.ShaderCompiler; +using SharpEmu.ShaderCompiler.Resources; +using SharpEmu.ShaderCompiler.Vulkan; +using Xunit; +using static SharpEmu.ShaderCompiler.Tests.Resources.ResourceTestProgram; + +namespace SharpEmu.Libs.Tests.Gpu.Vulkan; + +public sealed class MoveRelativeRegisterDeviceTests(HeadlessVulkanFixture fixture) : IClassFixture +{ + [Theory] + [InlineData(31u)] + [InlineData(100u)] + [InlineData(480u)] + public void DynamicWriteThenRead_PreservesRegistersNotNamedByTheProgram(uint offset) + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) return; + var program = Program( + MoveScalarRegister(0, 124, 8), + Vop1(4, "VMovreldB32", 5, Gen5Operand.Scalar(9)), + Vop1(8, "VMovrelsB32", 2, Gen5Operand.Vector(5)), + BufferAccess(12, "BufferStoreDword", 4, vectorData: 2), + EndProgram(20)); + var (plan, resources, layout) = Prepare(program); + var request = new ShaderCompileRequest(plan, resources, layout) { LocalSizeX = 1, ThreadCountX = 1 }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + using var harness = new ImageTestHarness(vulkan!); + using var runner = new LayoutComputeRunner(harness, request, shader.Spirv); + var result = runner.CreateBuffer(64); + var registers = new uint[256]; + registers[6] = 64; + registers[8] = offset; + registers[9] = 0x1234_5678; + harness.Run(() => runner.Dispatch(registers, + new Dictionary { [DescriptorBindingKind.Buffers] = [result] }, 1)); + Assert.Equal(registers[9], BinaryPrimitives.ReadUInt32LittleEndian(runner.ReadBack(result, 0, sizeof(uint)))); + harness.AssertNoValidationMessages(); + } +} diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/NativeHalfConversionProbeDeviceTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/NativeHalfConversionProbeDeviceTests.cs new file mode 100644 index 0000000..61f9648 --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/NativeHalfConversionProbeDeviceTests.cs @@ -0,0 +1,64 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.Libs.Gpu.Vulkan; +using SharpEmu.Libs.Tests.Gpu.Images; +using Silk.NET.Vulkan; +using Xunit; +using Xunit.Abstractions; + +namespace SharpEmu.Libs.Tests.Gpu.Vulkan; + +public sealed class NativeHalfConversionProbeDeviceTests(HeadlessVulkanFixture fixture, ITestOutputHelper output) + : IClassFixture +{ + // The probe is what decides whether the translator may use GLSL UnpackHalf2x16 / + // PackHalf2x16, so the gate is that it completes and reports a count - not that this + // device passes it. The verdict goes to the test output, where a port to another GPU + // can read it. + [Fact] + public void ProbeRunsAndReportsWhetherNativeHalfConversionIsExact() + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan)) return; + using var harness = new ImageTestHarness(vulkan); + + var result = default(NativeHalfConversionProbe.Result); + harness.Run(() => result = NativeHalfConversionProbe.Run( + harness.Device, + harness.Scheduler, + () => new CommandBuffer(harness.Scheduler.Current.Handle))); + + output.WriteLine( + $"[VK][F16] native half conversion exact={result.Exact} mismatches={result.Mismatches} " + + $"tested={result.Tested} device={vulkan.DeviceName} first={result.Detail}"); + Assert.Null(result.Note); + Assert.True(result.Tested >= 2 * 0x1_0000, $"tested={result.Tested}"); + Assert.InRange(result.Mismatches, 0, result.Tested); + Assert.Equal(result.Mismatches == 0, result.Exact); + harness.AssertNoValidationMessages(); + } + + // Every test vector must agree with System.Half, which is what proves the integer + // sequences the probe compares against are the right reference in the first place. + [Fact] + public void ReferenceConversionsMatchSystemHalfForEveryTestVector() + { + foreach (var value in HalfConversionReference.BuildTestVectors()) + { + var half = BitConverter.UInt16BitsToHalf((ushort)value); + var widened = BitConverter.SingleToUInt32Bits((float)half); + var wide = HalfConversionReference.HalfToFloatBits(value & 0xFFFFu); + Assert.True( + HalfConversionReference.Matches(widened, wide, asHalf: false), + $"half 0x{value & 0xFFFF:X4} widened to 0x{wide:X8}, System.Half says 0x{widened:X8}"); + + var single = BitConverter.UInt32BitsToSingle(value); + var narrowed = BitConverter.HalfToUInt16Bits((Half)single); + var narrow = HalfConversionReference.FloatToHalfBits(value); + Assert.True( + HalfConversionReference.Matches(narrowed, narrow, asHalf: true), + $"float 0x{value:X8} narrowed to 0x{narrow:X4}, System.Half says 0x{narrowed:X4}"); + } + } +} diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/PackedD16BufferTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/PackedD16BufferTests.cs new file mode 100644 index 0000000..a9f081b --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/PackedD16BufferTests.cs @@ -0,0 +1,88 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Buffers.Binary; +using SharpEmu.Libs.Gpu.Buffers; +using SharpEmu.Libs.Tests.Gpu.Images; +using SharpEmu.ShaderCompiler; +using SharpEmu.ShaderCompiler.Resources; +using SharpEmu.ShaderCompiler.Vulkan; +using Xunit; +using Xunit.Abstractions; +using static SharpEmu.ShaderCompiler.Tests.Resources.ResourceTestProgram; + +namespace SharpEmu.Libs.Tests.Gpu.Vulkan; + +public sealed class PackedD16BufferTests(HeadlessVulkanFixture fixture, ITestOutputHelper output) : IClassFixture +{ + [Theory] + [InlineData(false, false, 75u)] + [InlineData(false, true, 75u)] + [InlineData(true, false, 75u)] + [InlineData(false, false, 76u)] + [InlineData(false, true, 76u)] + [InlineData(true, false, 76u)] + [InlineData(false, false, 77u)] + [InlineData(false, true, 77u)] + [InlineData(true, false, 77u)] + public void BoundD16_LoadAndStoreUsePackedRegisters(bool typed, bool generic, uint format) + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan)) return; + Gen5ShaderInstruction Access(uint pc, bool store, int offset) + { + var instruction = BufferAccess(pc, (typed ? "TBuffer" : "Buffer") + + (store ? "Store" : "Load") + "FormatD16Xyzw", 0, offset, 2, vectorData: 4); + return instruction with + { + Encoding = typed ? Gen5ShaderEncoding.Mtbuf : Gen5ShaderEncoding.Mubuf, + Control = ((Gen5BufferMemoryControl)instruction.Control!) with + { Typed = typed, TypedFormat = typed ? format : 0, PackedD16 = true, FormatComponentCount = 4 }, + }; + } + var program = Program(MoveVector(0, 6, 0xAABBCCDD), MoveVector(8, 7, 0x11223344), + Access(16, false, 0), BufferAccess(24, "BufferStoreDwordx4", 0, 0x800, 4, vectorData: 4), + Access(32, true, 0x100), EndProgram(40)); + var plan = ShaderResourcePlan.Extract(program, ShaderStage.Compute, 1, 0, 16); + var registers = new uint[16]; + registers[1] = 2; registers[2] = 0x1000; + registers[3] = DescriptorConstants.IdentityDestinationSelect | (format << 12); + var snapshot = new ResourceSnapshot(); + var specialization = new ResourceSpecialization(); + static bool NoMemory(ulong address, out uint word) { word = 0; return false; } + Assert.True(ResourceMaterializer.Materialize(plan, Inputs(registers, NoMemory, NoMemory), + ref snapshot, ref specialization, out var failure), failure.ToString()); + var resources = ResourceMaterializer.ApplyTo(plan, specialization); + var layout = BindingLayout.Allocate(resources.Info, BindingLayout.CollectUserDataRegisters(program, 0, 16), + false, ShaderCompileRequest.RequiresFlattenedTable(plan, resources), false); + var request = new ShaderCompileRequest(plan, resources, layout) + { ThreadCountX = 1, ThreadCountY = 1, ThreadCountZ = 1, ForceGenericBufferFormats = generic }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + using var harness = new ImageTestHarness(vulkan); + using var runner = new LayoutComputeRunner(harness, request, shader.Spirv); + var bytes = new byte[0x1000]; + uint[] input = format == 77 ? [0x3F800000u, 0xBF000000u, 0x40400000u, 0x40800000u] + : [0x00011234u, 0xFFFF8000u, 0x0002ABCDu, 0xFFFF7FFFu]; + for (var i = 0; i < 4; i++) BinaryPrimitives.WriteUInt32LittleEndian(bytes.AsSpan(i * 4), input[i]); + var buffer = runner.CreateBuffer(bytes); + var bindings = new Dictionary + { [DescriptorBindingKind.Buffers] = Enumerable.Repeat(buffer, resources.Info.Buffers.Count).ToArray() }; + harness.Run(() => runner.Dispatch(registers, bindings, 1)); + var result = harness.ReadBack(buffer.Handle, 0, (ulong)bytes.Length); + for (var i = 0; i < 4; i++) + { + var half = format == 77 ? BitConverter.HalfToUInt16Bits((Half)BitConverter.UInt32BitsToSingle(input[i])) + : input[i] & 0xFFFF; + var packed = BinaryPrimitives.ReadUInt32LittleEndian(result.AsSpan(0x800 + i / 2 * 4)); + Assert.Equal(half, (packed >> ((i & 1) * 16)) & 0xFFFF); + var stored = format == 77 ? BitConverter.SingleToUInt32Bits((float)BitConverter.UInt16BitsToHalf((ushort)half)) + : format == 76 ? unchecked((uint)(int)(short)half) : half; + Assert.Equal(stored, BinaryPrimitives.ReadUInt32LittleEndian(result.AsSpan(0x100 + i * 4))); + } + Assert.Equal(0xAABBCCDDu, BinaryPrimitives.ReadUInt32LittleEndian(result.AsSpan(0x808))); + Assert.Equal(0x11223344u, BinaryPrimitives.ReadUInt32LittleEndian(result.AsSpan(0x80C))); + harness.AssertNoValidationMessages(); + vulkan.AssertNoValidationMessages(); + output.WriteLine($"Verified D16 load/store typed={typed} generic={generic} format={format} on {vulkan.DeviceName}"); + } +} diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/PackedF16LiteralDeviceTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/PackedF16LiteralDeviceTests.cs new file mode 100644 index 0000000..14df1c8 --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/PackedF16LiteralDeviceTests.cs @@ -0,0 +1,65 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.Libs.Gpu.Buffers; +using SharpEmu.Libs.Tests.Gpu.Images; +using SharpEmu.ShaderCompiler; +using SharpEmu.ShaderCompiler.Resources; +using SharpEmu.ShaderCompiler.Tests; +using SharpEmu.ShaderCompiler.Vulkan; +using Xunit; +using static SharpEmu.ShaderCompiler.Tests.Resources.ResourceTestProgram; + +namespace SharpEmu.Libs.Tests.Gpu.Vulkan; + +// A 32-bit literal in a packed f16 (VOP3P) op holds two f16 halves that op_sel / op_sel_hi +// pick per lane, like a register. The instruction words are from Ghost of Yotei's TAA resolve +// (0xD42889155B0A6B9F), where reading the literal as one f32 turned the image black. +public sealed class PackedF16LiteralDeviceTests(HeadlessVulkanFixture fixture) : IClassFixture +{ + [Fact] + public void LiteralHalvesFollowOperandSelect() + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) return; + var program = Program( + MoveVectorFromScalar(0, 6, 8), + MoveVectorFromScalar(4, 63, 9), + MoveVectorFromScalar(8, 62, 10), + // v_pk_fma_f16 v0, 0x34003800, v6, v63 op_sel:[1,1,0] op_sel_hi:[0,1,1] neg_hi:[0,0,1] + Vop3p(12, 0xCC0E5C00u, 0x14FE0CFFu, 0x34003800u), + // v_pk_min_f16 v4, 0x77FF, v62 op_sel_hi:[0,1] + Vop3p(24, 0xCC114004u, 0x10027CFFu, 0x000077FFu), + Vop2(36, "VOrB32", 20, Gen5Operand.Vector(0), Gen5Operand.Vector(0)), + Vop2(40, "VOrB32", 21, Gen5Operand.Vector(4), Gen5Operand.Vector(4)), + Vop2(44, "VAndB32", 1, Operand(0), Gen5Operand.Vector(1)), + BufferAccess(48, "BufferStoreDwordx2", 4, dwords: 2, vectorData: 20, offsetEnabled: true, vectorAddress: 1), + EndProgram(56)); + var (plan, resources, layout) = Prepare(program); + var request = new ShaderCompileRequest(plan, resources, layout) { LocalSizeX = 64 }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + using var harness = new ImageTestHarness(vulkan); + using var runner = new LayoutComputeRunner(harness, request, shader.Spirv); + var output = runner.CreateBuffer(8); + var bindings = new Dictionary + { + [DescriptorBindingKind.Buffers] = Enumerable.Repeat(output, Math.Max(1, plan.Info.Buffers.Count)).ToArray(), + }; + var registers = new uint[256]; + registers[6] = 8; + registers[8] = 0x4000_0000; // v6: high 2.0, low 0 + registers[9] = 0x3C00_3C00; // v63: 1.0 in both halves + registers[10] = 0x4000_7C00; // v62: high 2.0, low +Inf + harness.Run(() => runner.Dispatch(registers, bindings, 1)); + harness.AssertNoValidationMessages(); + var bytes = runner.ReadBack(output, 0, 8); + // Low lane: 0.25 (literal high half) * 2 + 1 = 1.5; high lane: 0.5 (literal low half) * 2 - 1 = 0. + // Read as one f32 (1.2e-7 in both lanes) the lanes would be 1 and -1. + Assert.Equal(0x0000_3E00u, BitConverter.ToUInt32(bytes, 0)); + // Both lanes read the literal's low half, 32752: min(32752, +Inf), min(32752, 2). + Assert.Equal(0x4000_77FFu, BitConverter.ToUInt32(bytes, 4)); + } + + private static Gen5ShaderInstruction Vop3p(uint pc, uint word0, uint word1, uint literal) => + Gen5Float16ArithmeticTests.Decode([word0, word1, literal, 0xBF81_0000u]).Instructions[0] with { Pc = pc }; +} diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/RuntimeBufferStrideTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/RuntimeBufferStrideTests.cs new file mode 100644 index 0000000..293b2ef --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/RuntimeBufferStrideTests.cs @@ -0,0 +1,102 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Buffers.Binary; +using SharpEmu.Libs.Gpu.Buffers; +using SharpEmu.Libs.Tests.Gpu.Images; +using SharpEmu.ShaderCompiler; +using SharpEmu.ShaderCompiler.Resources; +using SharpEmu.ShaderCompiler.Tests.Resources; +using SharpEmu.ShaderCompiler.Vulkan; +using Xunit; +using Xunit.Abstractions; +using static SharpEmu.ShaderCompiler.Tests.Resources.ResourceTestProgram; + +namespace SharpEmu.Libs.Tests.Gpu.Vulkan; + +// A buffer's stride reaches the shader as data, so the same compiled program indexes +// correctly with every stride a title binds, and the stride is not part of the key. +public sealed class RuntimeBufferStrideTests(HeadlessVulkanFixture fixture, ITestOutputHelper output) : IClassFixture +{ + private const ulong BufferAddress = 0x2_0000_0000; + private const int BufferBytes = 0x1000; + private const int ResultOffset = 0x800; + private const uint Index = 3; + + // v6 = s8; v4 = buffer[s0:3] at v6 * stride; buffer[0x800] = v4. + private static Gen5ShaderProgram IndexedLoadProgram() => Program( + MoveVectorFromScalar(0, 6, 8), + BufferAccess(4, "BufferLoadDword", 0, dwords: 1, vectorData: 4, indexEnabled: true, vectorAddress: 6), + BufferAccess(12, "BufferStoreDword", 0, offset: ResultOffset, dwords: 1, vectorData: 4), + EndProgram(20)); + + private static uint[] UserData(uint stride) + { + var registers = new uint[20]; + registers[0] = unchecked((uint)BufferAddress); + registers[1] = (uint)((BufferAddress >> 32) & 0xFFFF) | (stride << 16); + registers[2] = stride == 0 ? BufferBytes : BufferBytes / stride; + registers[3] = DescriptorConstants.IdentityDestinationSelect | (20u << 12); + registers[8] = Index; + return registers; + } + + private static ResourceSpecialization Specialize(ShaderResourcePlan plan, uint stride) + { + var snapshot = new ResourceSnapshot(); + var specialization = new ResourceSpecialization(); + static bool NoMemory(ulong address, out uint value) { value = 0; return false; } + var inputs = ResourceTestProgram.Inputs(UserData(stride), NoMemory, NoMemory); + Assert.True(ResourceMaterializer.Materialize(plan, inputs, ref snapshot, ref specialization, out var failure), failure.ToString()); + Assert.Equal(stride, Assert.Single(specialization.Buffers).PackedStride & BufferSpecialization.StrideMask); + for (var index = 0; index < specialization.Buffers.Count; index++) + { + specialization.Buffers[index] = specialization.Buffers[index].WithoutRuntimeStride(); + } + + return specialization; + } + + [Fact] + public void OneProgram_IndexesWithEveryBoundStride() + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan)) return; + + var plan = ShaderResourcePlan.Extract(IndexedLoadProgram(), ShaderStage.Compute, 1, 0, 20); + var specialization = Specialize(plan, 8); + // Another stride is the same permutation. + Assert.Equal(specialization, Specialize(plan, 20)); + + var resources = ResourceMaterializer.ApplyTo(plan, specialization); + var layout = BindingLayout.Allocate(resources.Info, + BindingLayout.CollectUserDataRegisters(plan.Graph.Program, 0, 20), false, + ShaderCompileRequest.RequiresFlattenedTable(plan, resources), false, + usesRuntimeBufferStrides: true); + Assert.True(layout.UsesRuntimeBufferStrides); + var request = new ShaderCompileRequest(plan, resources, layout) { ThreadCountX = 1, ThreadCountY = 1, ThreadCountZ = 1 }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + + using var harness = new ImageTestHarness(vulkan); + using var runner = new LayoutComputeRunner(harness, request, shader.Spirv); + foreach (var stride in new uint[] { 8, 20 }) + { + var initial = new byte[BufferBytes]; + for (var word = 0; word < ResultOffset / 4; word++) + { + BinaryPrimitives.WriteUInt32LittleEndian(initial.AsSpan(word * 4), 0xA000_0000u + (uint)word); + } + + var records = runner.CreateBuffer(initial); + var bindings = new Dictionary { [DescriptorBindingKind.Buffers] = [records] }; + harness.Run(() => runner.Dispatch(UserData(stride), bindings, 1, bufferStrides: [stride])); + var result = harness.ReadBack(records.Handle, 0, BufferBytes); + var expected = BinaryPrimitives.ReadUInt32LittleEndian(initial.AsSpan((int)(Index * stride))); + Assert.Equal(expected, BinaryPrimitives.ReadUInt32LittleEndian(result.AsSpan(ResultOffset))); + output.WriteLine($"stride={stride} loaded 0x{expected:X8} on {vulkan.DeviceName}"); + } + + harness.AssertNoValidationMessages(); + vulkan.AssertNoValidationMessages(); + } +} diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/ScalarAbsoluteDeviceTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/ScalarAbsoluteDeviceTests.cs index 1e504e5..7931424 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/ScalarAbsoluteDeviceTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/ScalarAbsoluteDeviceTests.cs @@ -15,6 +15,38 @@ namespace SharpEmu.Libs.Tests.Gpu.Vulkan; public sealed class ScalarAbsoluteDeviceTests(HeadlessVulkanFixture fixture) : IClassFixture { + [Fact] + public void LeadingBitCountsUnsignedZerosAndPreservesCondition() + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) return; + var program = Program( + Sop2(0, "SCmpEqI32", 0, Gen5Operand.Scalar(8), Gen5Operand.Scalar(8)) with { Encoding = Gen5ShaderEncoding.Sopc, Destinations = [] }, + Sop1(4, "SFlbitI32B32", 10, Gen5Operand.Scalar(8)), + MoveVectorFromScalar(8, 12, 10), + Sop2(12, "SCselectB32", 11, Operand(1), Operand(0)), + MoveVectorFromScalar(16, 13, 11), + BufferAccess(20, "BufferStoreDwordx2", 4, dwords: 2, vectorData: 12), EndProgram(28)); + var (plan, resources, layout) = Prepare(program); + var request = new ShaderCompileRequest(plan, resources, layout) { LocalSizeX = 1, ThreadCountX = 1 }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + using var harness = new ImageTestHarness(vulkan); + using var runner = new LayoutComputeRunner(harness, request, shader.Spirv); + var result = runner.CreateBuffer(64); + var bindings = new Dictionary { [DescriptorBindingKind.Buffers] = [result] }; + var registers = new uint[256]; + registers[6] = 64; + foreach (var (input, expected) in new (uint, uint)[] { (0, uint.MaxValue), (0xCCCC, 16), (0xFFFF3333, 0), (0x7FFFFFFF, 1), (0x80000000, 0), (uint.MaxValue, 0) }) + { + registers[8] = input; + harness.Run(() => runner.Dispatch(registers, bindings, 1)); + var actual = runner.ReadBack(result, 0, 8); + Assert.Equal(expected, BinaryPrimitives.ReadUInt32LittleEndian(actual)); + Assert.Equal(1u, BinaryPrimitives.ReadUInt32LittleEndian(actual.AsSpan(4))); + } + harness.AssertNoValidationMessages(); + } + [Theory] [InlineData(false, false, false)] [InlineData(false, true, true)] diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/ScalarPairBitDeviceTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/ScalarPairBitDeviceTests.cs new file mode 100644 index 0000000..9afae55 --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/ScalarPairBitDeviceTests.cs @@ -0,0 +1,72 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Buffers.Binary; +using System.Numerics; +using SharpEmu.Libs.Gpu.Buffers; +using SharpEmu.Libs.Tests.Gpu.Images; +using SharpEmu.ShaderCompiler; +using SharpEmu.ShaderCompiler.Resources; +using SharpEmu.ShaderCompiler.Vulkan; +using Xunit; +using static SharpEmu.ShaderCompiler.Tests.Resources.ResourceTestProgram; + +namespace SharpEmu.Libs.Tests.Gpu.Vulkan; + +public sealed class ScalarPairBitDeviceTests(HeadlessVulkanFixture fixture) : IClassFixture +{ + public static IEnumerable Inputs() + { + foreach (var opcode in new[] { "SBcnt1I32B64", "SFlbitI32B64" }) + foreach (uint destination in new uint[] { 20, 106 }) + { + yield return [opcode, 0ul, destination]; + yield return [opcode, ulong.MaxValue, destination]; + yield return [opcode, 0x5555555555555555ul, destination]; + yield return [opcode, 0xAAAAAAAAAAAAAAAAul, destination]; + for (var bit = 0; bit < 64; bit++) yield return [opcode, 1ul << bit, destination]; + } + } + + [Theory] + [MemberData(nameof(Inputs))] + public void PairBitOperationPreservesTheNextRegisterAndSetsTheExpectedCondition( + string opcode, ulong input, uint destination) + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) return; + var (plan, resources, layout) = Prepare(Program( + MoveScalar(0, destination, (uint)input), + MoveScalar(4, destination + 1, (uint)(input >> 32)), + Sopc(8, "SCmpEqU32", Operand(1u), Operand(1u)), + Sop1(12, opcode, destination, Gen5Operand.Scalar(destination)), + MoveVectorFromScalar(16, 2, destination), + MoveVectorFromScalar(20, 3, destination + 1), + Sop2(24, "SCselectB32", 22, Operand(1u), Operand(0u)), + MoveVectorFromScalar(28, 4, 22), + BufferAccess(32, "BufferStoreDwordx2", 4, 0, 2, 2), + BufferAccess(40, "BufferStoreDword", 4, 8, 1, 4), + EndProgram(48))); + var request = new ShaderCompileRequest(plan, resources, layout) + { + LocalSizeX = 1, ThreadCountX = 1, WaveSize = 32, + }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + using var harness = new ImageTestHarness(vulkan); + using var runner = new LayoutComputeRunner(harness, request, shader.Spirv); + var output = runner.CreateBuffer(256); + var registers = new uint[256]; + registers[6] = 256; + harness.Run(() => runner.Dispatch(registers, + new Dictionary { [DescriptorBindingKind.Buffers] = [output] }, 1)); + var bytes = runner.ReadBack(output, 0, 12); + var expected = opcode == "SBcnt1I32B64" + ? (uint)BitOperations.PopCount(input) + : input == 0 ? uint.MaxValue : (uint)BitOperations.LeadingZeroCount(input); + Assert.Equal(expected, BinaryPrimitives.ReadUInt32LittleEndian(bytes)); + Assert.Equal((uint)(input >> 32), BinaryPrimitives.ReadUInt32LittleEndian(bytes.AsSpan(4))); + Assert.Equal(opcode == "SBcnt1I32B64" && input == 0 ? 0u : 1u, + BinaryPrimitives.ReadUInt32LittleEndian(bytes.AsSpan(8))); + harness.AssertNoValidationMessages(); + } +} diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/SdwaFloatModifierDeviceTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/SdwaFloatModifierDeviceTests.cs new file mode 100644 index 0000000..98a5afe --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/SdwaFloatModifierDeviceTests.cs @@ -0,0 +1,70 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.Libs.Gpu.Buffers; +using SharpEmu.Libs.Tests.Gpu.Images; +using SharpEmu.ShaderCompiler; +using SharpEmu.ShaderCompiler.Resources; +using SharpEmu.ShaderCompiler.Tests; +using SharpEmu.ShaderCompiler.Vulkan; +using Xunit; +using static SharpEmu.ShaderCompiler.Tests.Resources.ResourceTestProgram; + +namespace SharpEmu.Libs.Tests.Gpu.Vulkan; + +// Float VOP2 SDWA with DWORD selects, used only for its source negates, output modifier and +// clamp. The instruction words are taken from Ghost of Yotei's TAA resolve (0xD42889155B0A6B9F). +public sealed class SdwaFloatModifierDeviceTests(HeadlessVulkanFixture fixture) : IClassFixture +{ + [Fact] + public void NegateOutputModifierAndClampApply() + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) return; + const float x = 1.5f, y = 0.25f, z = 3f, w = -2f; + var program = Program( + MoveVectorFromScalar(0, 42, 8), + MoveVectorFromScalar(4, 41, 9), + MoveVectorFromScalar(8, 12, 10), + MoveVectorFromScalar(12, 2, 11), + MoveVectorFromScalar(16, 15, 8), + MoveVectorFromScalar(20, 11, 10), + // v_add_f32_sdwa v4, -v42, -v41 + Sdwa(24, 0x060852F9u, 0x1616062Au), + // v_mul_f32_sdwa v13, v12, v2 div:2 + Sdwa(32, 0x101A04F9u, 0x0606C60Cu), + // v_mul_f32_sdwa v1, v15, v11 clamp + Sdwa(40, 0x100216F9u, 0x0606260Fu), + Vop2(48, "VOrB32", 20, Gen5Operand.Vector(4), Gen5Operand.Vector(4)), + Vop2(52, "VOrB32", 21, Gen5Operand.Vector(13), Gen5Operand.Vector(13)), + Vop2(56, "VOrB32", 22, Gen5Operand.Vector(1), Gen5Operand.Vector(1)), + Vop2(60, "VAndB32", 1, Operand(0), Gen5Operand.Vector(1)), + BufferAccess(64, "BufferStoreDwordx3", 4, dwords: 3, vectorData: 20, offsetEnabled: true, vectorAddress: 1), + EndProgram(72)); + var (plan, resources, layout) = Prepare(program); + var request = new ShaderCompileRequest(plan, resources, layout) { LocalSizeX = 64 }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + using var harness = new ImageTestHarness(vulkan); + using var runner = new LayoutComputeRunner(harness, request, shader.Spirv); + var output = runner.CreateBuffer(16); + var bindings = new Dictionary + { + [DescriptorBindingKind.Buffers] = Enumerable.Repeat(output, Math.Max(1, plan.Info.Buffers.Count)).ToArray(), + }; + var registers = new uint[256]; + registers[6] = 16; + registers[8] = BitConverter.SingleToUInt32Bits(x); + registers[9] = BitConverter.SingleToUInt32Bits(y); + registers[10] = BitConverter.SingleToUInt32Bits(z); + registers[11] = BitConverter.SingleToUInt32Bits(w); + harness.Run(() => runner.Dispatch(registers, bindings, 1)); + harness.AssertNoValidationMessages(); + var bytes = runner.ReadBack(output, 0, 12); + Assert.Equal(-(x + y), BitConverter.ToSingle(bytes, 0)); + Assert.Equal(z * w * 0.5f, BitConverter.ToSingle(bytes, 4)); + Assert.Equal(1f, BitConverter.ToSingle(bytes, 8)); + } + + private static Gen5ShaderInstruction Sdwa(uint pc, uint word0, uint word1) => + Gen5Float16ArithmeticTests.Decode([word0, word1, 0xBF81_0000u]).Instructions[0] with { Pc = pc }; +} diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/StaticBufferFormatDeviceTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/StaticBufferFormatDeviceTests.cs new file mode 100644 index 0000000..de140e4 --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/StaticBufferFormatDeviceTests.cs @@ -0,0 +1,104 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.Libs.Gpu.Buffers; +using SharpEmu.Libs.Tests.Gpu.Images; +using SharpEmu.ShaderCompiler; +using SharpEmu.ShaderCompiler.Resources; +using SharpEmu.ShaderCompiler.Vulkan; +using Xunit; +using static SharpEmu.ShaderCompiler.Tests.Resources.ResourceTestProgram; + +namespace SharpEmu.Libs.Tests.Gpu.Vulkan; + +// A formatted buffer load whose descriptor format is known when the shader is translated +// is emitted for that format only. On the device it must give the run-time decoder's bits +// for every unified format, with a swizzle, and past the end of the buffer. +public sealed class StaticBufferFormatDeviceTests(HeadlessVulkanFixture fixture) : IClassFixture +{ + private const uint Threads = 256; + private const uint Stride = 16; + + // Identity, and W, X, one, zero. + private static readonly uint[] Swizzles = + [ + 4u | (5u << 3) | (6u << 6) | (7u << 9), + 7u | (4u << 3) | (1u << 6) | (0u << 9), + ]; + + public static IEnumerable Formats() + { + for (uint format = 1; format < 128; format++) + { + if (Gfx10UnifiedFormat.TryDecode(format, out _, out _)) + { + yield return [format]; + } + } + } + + [Theory] + [MemberData(nameof(Formats))] + public void StaticFormatLoadMatchesTheRuntimeDecoder(uint format) + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) return; + var data = ImageTestHarness.Pattern((int)(Threads * Stride), seed: format); + foreach (var swizzle in Swizzles) + { + var generic = Run(vulkan, data, format, swizzle, forceGeneric: true); + var specialized = Run(vulkan, data, format, swizzle, forceGeneric: false); + for (var offset = 0; offset < generic.Length; offset += 4) + { + var expected = BitConverter.ToUInt32(generic, offset); + var actual = BitConverter.ToUInt32(specialized, offset); + Assert.True(expected == actual, + $"format={format} swizzle=0x{swizzle:X3} thread={offset / 16} component={offset % 16 / 4} " + + $"runtime=0x{expected:X8} static=0x{actual:X8}"); + } + } + } + + // v1 = v0 * 16; v4..v7 = format_load(s[8:11], v1); store v4..v7 to s[4:7] at v1. + private static byte[] Run(HeadlessVulkan vulkan, byte[] data, uint format, uint swizzle, bool forceGeneric) + { + var program = Program( + Vop2(0, "VLshlrevB32", 1, Operand(4), Gen5Operand.Vector(0)), + BufferAccess(4, "BufferLoadFormatXyzw", 8, 0, 4, vectorData: 4, offsetEnabled: true, vectorAddress: 1), + BufferAccess(12, "BufferStoreDwordx4", 4, 0, 4, vectorData: 4, offsetEnabled: true, vectorAddress: 1), + EndProgram(20)); + var (plan, resources, layout) = Prepare(program); + var loadResource = MemoryAccessInfo.NoResource; + for (var index = 0; index < plan.Memory.Count; index++) + { + if (plan.Memory[index].Opcode == "BufferLoadFormatXyzw") loadResource = plan.Memory[index].Resource; + } + + resources.Info.Buffers[(int)loadResource].DescriptorFormat = format; + resources.Info.Buffers[(int)loadResource].DescriptorSwizzle = swizzle; + var request = new ShaderCompileRequest(plan, resources, layout) + { + LocalSizeX = Threads, + ForceGenericBufferFormats = forceGeneric, + }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + using var harness = new ImageTestHarness(vulkan); + using var runner = new LayoutComputeRunner(harness, request, shader.Spirv); + var input = runner.CreateBuffer(data); + var output = runner.CreateBuffer(Threads * Stride); + var buffers = new GpuBuffer[Math.Max(resources.Info.Buffers.Count, 2)]; + for (var index = 0; index < buffers.Length; index++) + { + buffers[index] = index == (int)loadResource ? input : output; + } + + var registers = new uint[256]; + registers[6] = Threads * Stride; + // The last few elements are past num_records and read the out-of-range value. + registers[10] = Threads * Stride - 40; + registers[11] = (format << 12) | swizzle; + harness.Run(() => runner.Dispatch(registers, new Dictionary { [DescriptorBindingKind.Buffers] = buffers }, 1)); + harness.AssertNoValidationMessages(); + return runner.ReadBack(output, 0, Threads * Stride); + } +} diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/StructuredControlFlowDeviceTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/StructuredControlFlowDeviceTests.cs new file mode 100644 index 0000000..1d504f9 --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/StructuredControlFlowDeviceTests.cs @@ -0,0 +1,83 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Buffers.Binary; +using SharpEmu.Libs.Gpu.Buffers; +using SharpEmu.Libs.Tests.Gpu.Images; +using SharpEmu.ShaderCompiler; +using SharpEmu.ShaderCompiler.Resources; +using SharpEmu.ShaderCompiler.Tests; +using SharpEmu.ShaderCompiler.Vulkan; +using Xunit; +using static SharpEmu.ShaderCompiler.Tests.Resources.ResourceTestProgram; + +namespace SharpEmu.Libs.Tests.Gpu.Vulkan; + +// Runs each control-flow program through both the structured form and the PC dispatcher on the +// device, and checks both against the reference model. +public sealed class StructuredControlFlowDeviceTests(HeadlessVulkanFixture fixture) : IClassFixture +{ + public static IEnumerable Cases() + { + foreach (var name in new[] + { + nameof(Gen5StructuredControlFlowTests.LoopWithIfElse), + nameof(Gen5StructuredControlFlowTests.NestedLoopsWithEarlyExit), + nameof(Gen5StructuredControlFlowTests.NestedIfSharedMerge), + nameof(Gen5StructuredControlFlowTests.IrreducibleCycle), + }) + { + yield return [name]; + } + } + + private static readonly (uint Input0, uint Input1)[] Inputs = + [ + (0, 0), (1, 0), (2, 1), (3, 3), (4, 6), (5, 7), (6, 9), (7, 20), (8, 1000), (9, 16), + ]; + + [Theory] + [MemberData(nameof(Cases))] + public void StructuredAndDispatcherFormsMatchTheModel(string name) + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) return; + foreach (var forceDispatcher in new[] { false, true }) + { + var (plan, resources, layout) = Prepare(Gen5StructuredControlFlowTests.Create(name)); + var request = new ShaderCompileRequest(plan, resources, layout) + { + LocalSizeX = 1, + ThreadCountX = 1, + ForceDispatcher = forceDispatcher, + }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + using var harness = new ImageTestHarness(vulkan); + using var runner = new LayoutComputeRunner(harness, request, shader.Spirv); + var output = runner.CreateBuffer(64); + // Every store names the same V#; the planner may still give each path its own binding. + var bindings = new Dictionary + { + [DescriptorBindingKind.Buffers] = Enumerable.Repeat(output, Math.Max(1, plan.Info.Buffers.Count)).ToArray(), + }; + var registers = new uint[256]; + registers[6] = 64; + foreach (var (input0, input1) in Inputs) + { + registers[8] = input0; + registers[9] = input1; + harness.Run(() => runner.Dispatch(registers, bindings, 1)); + var actual = runner.ReadBack(output, 0, 8); + var expected = Gen5StructuredControlFlowTests.Expected(name, input0, input1); + var form = forceDispatcher ? "dispatcher" : "structured"; + Assert.True( + expected.Acc == BinaryPrimitives.ReadUInt32LittleEndian(actual) && + expected.Second == BinaryPrimitives.ReadUInt32LittleEndian(actual.AsSpan(4)), + $"{name} {form} inputs=({input0},{input1}) expected=({expected.Acc},{expected.Second}) " + + $"actual=({BinaryPrimitives.ReadUInt32LittleEndian(actual)},{BinaryPrimitives.ReadUInt32LittleEndian(actual.AsSpan(4))})"); + } + + harness.AssertNoValidationMessages(); + } + } +} diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/TessellationWaveDeviceTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/TessellationWaveDeviceTests.cs new file mode 100644 index 0000000..d314424 --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/TessellationWaveDeviceTests.cs @@ -0,0 +1,252 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Buffers.Binary; +using SharpEmu.Libs.Gpu.Buffers; +using SharpEmu.Libs.Tests.Gpu.Images; +using SharpEmu.ShaderCompiler; +using SharpEmu.ShaderCompiler.Resources; +using SharpEmu.ShaderCompiler.Vulkan; +using Xunit; +using static SharpEmu.ShaderCompiler.Tests.Resources.ResourceTestProgram; + +namespace SharpEmu.Libs.Tests.Gpu.Vulkan; + +public sealed class TessellationWaveDeviceTests(HeadlessVulkanFixture fixture) : IClassFixture +{ + [Fact] + public void MergedHull_BackAddressIsUploadedAndSurvivesLocalUserRegisterReuse() + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) return; + var merged = Program( + MoveScalar(0, 8, 0xBAD), MoveScalar(4, 9, 0xBAD), + MoveVectorFromScalar(256, 10, 0), MoveVectorFromScalar(260, 11, 1), + Vop2(264, "VLshlrevB32", 12, Operand(3), Gen5Operand.Vector(3)), + BufferAccess(268, "BufferStoreDwordx2", 24, dwords: 2, vectorData: 10, offsetEnabled: true, vectorAddress: 12), + EndProgram(276)) with { FusedContinuationPc = 256 }; + var program = Gen5TessellationLowering.PrepareMergedHull(merged, 32); + var plan = ShaderResourcePlan.Extract(program, ShaderStage.Compute, Hash, 8, 26, waveSize: 64); + var resources = ResourceMaterializer.ApplyTo(plan, ResourceSpecialization.Default(plan.Info)); + var layout = BindingLayout.Allocate(resources.Info, BindingLayout.CollectUserDataRegisters(program, 8, 26), + false, false, false, usesTessellationData: true); + var request = new ShaderCompileRequest(plan, resources, layout) + { + LocalSizeX = 64, WaveSize = 64, CooperativeWave64Workgroup = true, + TessellationHull = new(4, 4, 16, program.FusedContinuationPc!.Value), + }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + SharpEmu.ShaderCompiler.Tests.Gen5LargeDispatcherValidationTests.ValidateWithSpirvToolsWhenAvailable(shader.Spirv); + using var harness = new ImageTestHarness(vulkan); + using var runner = new LayoutComputeRunner(harness, request, shader.Spirv); + var buffer = runner.CreateBuffer(512); + var scalar = new uint[34]; scalar[8] = 0x1111; scalar[9] = 0x20; scalar[26] = 512; + var data = new uint[Gen5TessellationData.DwordCount]; data[Gen5TessellationData.PatchCount] = 16; + foreach (ulong address in new[] { 0x20_123456BCul, 0x21_87654304ul }) + { + scalar[32] = (uint)address; scalar[33] = (uint)(address >> 32); + harness.Run(() => runner.Dispatch(scalar, + new Dictionary { [DescriptorBindingKind.Buffers] = [buffer] }, 1, + tessellationData: data)); + var bytes = runner.ReadBack(buffer, 0, 512); + for (var lane = 0; lane < 64; lane++) + Assert.Equal(address, BinaryPrimitives.ReadUInt64LittleEndian(bytes.AsSpan(lane * 8))); + } + harness.AssertNoValidationMessages(); + } + + [Theory] + [InlineData(128u)] + [InlineData(256u)] + public void MultipleWaves_CollectAllSixtyFourLaneBits(uint threads) + { + var values = Run(Program( + Vopc(0, "VCmpLeU32", Operand(0), 0), + MoveVectorFromScalar(4, 4, 106), + Vop2(8, "VLshlrevB32", 5, Operand(2), Gen5Operand.Vector(0)), + BufferAccess(12, "BufferStoreDword", 4, vectorData: 4, offsetEnabled: true, vectorAddress: 5), + EndProgram(20)), threads); + if (values is null) return; + Assert.All(values, value => Assert.Equal(uint.MaxValue, value)); + } + + [Fact] + public void IndependentlyBranchingWaves_PreserveTheirScalarAndVectorState() + { + var program = Program( + new Gen5ShaderInstruction(0, Gen5ShaderEncoding.Vop1, "VReadfirstlaneB32", [0u], + [Gen5Operand.Vector(0)], [Gen5Operand.Scalar(20)], null), + Sopc(4, "SCmpLtU32", Gen5Operand.Scalar(20), Operand(64)), + Branch(8, "SCbranchScc0", 5), // else at 32 + MoveScalar(12, 21, 111), + Vop1(16, "VMovB32", 4, Operand(333)), + Branch(20, "SBranch", 4), // join at 40 + Nop(24), Nop(28), + MoveScalar(32, 21, 222), + Vop1(36, "VMovB32", 4, Operand(444)), + Vop2(40, "VAddI32", 4, Gen5Operand.Scalar(21), Gen5Operand.Vector(4)), + Vop2(44, "VLshlrevB32", 5, Operand(2), Gen5Operand.Vector(0)), + BufferAccess(48, "BufferStoreDword", 4, vectorData: 4, offsetEnabled: true, vectorAddress: 5), + EndProgram(56)); + var values = Run(program, 256); + if (values is null) return; + for (var lane = 0; lane < values.Length; lane++) Assert.Equal(lane < 64 ? 444u : 666u, values[lane]); + } + + [Fact] + public void LocalAndHullPhases_ExchangeDataAcrossDifferentWaves() + { + var program = Program( + Vop2(0, "VLshlrevB32", 5, Operand(2), Gen5Operand.Vector(0)), + Vop2(4, "VAddI32", 4, Operand(37), Gen5Operand.Vector(0)), + DataShare(8, "DsWriteB32", false, [Gen5Operand.Vector(5), Gen5Operand.Vector(4)], []), + new Gen5ShaderInstruction(16, Gen5ShaderEncoding.Sopp, "SBarrier", [0u], [], [], null), + Vop2(20, "VXorB32", 6, Operand(64), Gen5Operand.Vector(0)), + Vop2(24, "VLshlrevB32", 6, Operand(2), Gen5Operand.Vector(6)), + DataShare(28, "DsReadB32", false, [Gen5Operand.Vector(6)], [4]), + BufferAccess(36, "BufferStoreDword", 4, vectorData: 4, offsetEnabled: true, vectorAddress: 5), + EndProgram(44)); + var values = Run(program, 256, 256); + if (values is null) return; + for (var lane = 0; lane < values.Length; lane++) Assert.Equal((uint)(lane ^ 64) + 37, values[lane]); + } + + // Null when the device cannot run the test. + private uint[]? Run(Gen5ShaderProgram program, uint threads, uint ldsDwords = 0) + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) return null; + var plan = ShaderResourcePlan.Extract(program, ShaderStage.Compute, Hash, 0, 16, waveSize: 64); + var registers = new uint[16]; registers[6] = threads * 4; + var resources = ResourceMaterializer.ApplyTo(plan, ResourceSpecialization.Default(plan.Info)); + var layout = BindingLayout.Allocate(resources.Info, BindingLayout.CollectUserDataRegisters(program, 0, 16), + false, ShaderCompileRequest.RequiresFlattenedTable(plan, resources), false); + var request = new ShaderCompileRequest(plan, resources, layout) + { LocalSizeX = threads, WaveSize = 64, CooperativeWave64Workgroup = true, LocalDataShareDwords = ldsDwords }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + SharpEmu.ShaderCompiler.Tests.Gen5LargeDispatcherValidationTests.ValidateWithSpirvToolsWhenAvailable(shader.Spirv); + using var harness = new ImageTestHarness(vulkan); + using var runner = new LayoutComputeRunner(harness, request, shader.Spirv); + var buffer = runner.CreateBuffer(threads * 4); + harness.Run(() => runner.Dispatch(registers, + new Dictionary { [DescriptorBindingKind.Buffers] = [buffer] }, 1)); + var bytes = runner.ReadBack(buffer, 0, threads * 4); + var values = new uint[threads]; + for (var lane = 0; lane < threads; lane++) values[lane] = BinaryPrimitives.ReadUInt32LittleEndian(bytes.AsSpan(lane * 4)); + harness.AssertNoValidationMessages(); + return values; + } + + [Fact] + public void MergedHullInputs_SplitPatchesNumberedAcrossInstances() + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) return; + var program = Program( + Vop1(0, "VMovB32", 10, Gen5Operand.Vector(1)), + Vop1(4, "VMovB32", 11, Gen5Operand.Vector(0)), + Vop1(8, "VMovB32", 12, Gen5Operand.Vector(2)), + Vop1(12, "VMovB32", 13, Gen5Operand.Vector(5)), + Vop2(16, "VLshlrevB32", 14, Operand(4), Gen5Operand.Vector(3)), + BufferAccess(20, "BufferStoreDwordx4", 24, dwords: 4, vectorData: 10, offsetEnabled: true, vectorAddress: 14), + EndProgram(28)); + var plan = ShaderResourcePlan.Extract(program, ShaderStage.Compute, Hash, 0, 64, waveSize: 64); + var resources = ResourceMaterializer.ApplyTo(plan, ResourceSpecialization.Default(plan.Info)); + var layout = BindingLayout.Allocate(resources.Info, BindingLayout.CollectUserDataRegisters(program, 0, 64), + false, false, false, usesTessellationData: true); + var request = new ShaderCompileRequest(plan, resources, layout) + { LocalSizeX = 256, WaveSize = 64, CooperativeWave64Workgroup = true, TessellationHull = new(4, 4, 63, 0) }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + using var harness = new ImageTestHarness(vulkan); + using var runner = new LayoutComputeRunner(harness, request, shader.Spirv); + var buffer = runner.CreateBuffer(4096); + var scalar = new uint[64]; scalar[26] = 4096; + var data = new uint[Gen5TessellationData.DwordCount]; + data[Gen5TessellationData.FirstPatch] = 7; + data[Gen5TessellationData.PatchCount] = 63; + data[Gen5TessellationData.VertexOffset] = 11; + data[Gen5TessellationData.InstanceId] = 100; + data[Gen5TessellationData.PatchesPerInstance] = 10; + harness.Run(() => runner.Dispatch(scalar, + new Dictionary { [DescriptorBindingKind.Buffers] = [buffer] }, 1, + tessellationData: data)); + var bytes = runner.ReadBack(buffer, 0, 4096); + for (uint lane = 0; lane < 252; lane++) + { + uint Read(int component) => BinaryPrimitives.ReadUInt32LittleEndian(bytes.AsSpan((int)lane * 16 + component * 4)); + var patch = 7 + lane / 4; + Assert.Equal((lane / 4) | ((lane % 4) << 8), Read(0)); + Assert.Equal(patch % 10, Read(1)); + Assert.Equal(patch % 10 * 4 + lane % 4 + 11, Read(2)); + Assert.Equal(100 + patch / 10, Read(3)); + } + harness.AssertNoValidationMessages(); + } + + [Theory] + [InlineData(0u, 0u)] + [InlineData(1u, 1u)] + [InlineData(2u, 2u)] + [InlineData(4u, 0u)] + public void MergedHullInputs_IncludePartialWaveCountsAndPatchControlPointIds(uint indexSize, uint byteBias) + { + var vulkan = fixture.Vulkan; + if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) return; + var program = Program( + MoveVectorFromScalar(0, 10, 3), + Vop1(4, "VMovB32", 11, Gen5Operand.Vector(0)), + Vop1(8, "VMovB32", 12, Gen5Operand.Vector(1)), + Vop1(12, "VMovB32", 13, Gen5Operand.Vector(2)), + Vop2(16, "VLshlrevB32", 14, Operand(4), Gen5Operand.Vector(3)), + BufferAccess(20, "BufferStoreDwordx4", 24, dwords: 4, vectorData: 10, offsetEnabled: true, vectorAddress: 14), + EndProgram(28)); + var plan = ShaderResourcePlan.Extract(program, ShaderStage.Compute, Hash, 0, 64, waveSize: 64); + var resources = ResourceMaterializer.ApplyTo(plan, ResourceSpecialization.Default(plan.Info)); + var layout = BindingLayout.Allocate(resources.Info, BindingLayout.CollectUserDataRegisters(program, 0, 64), + false, false, false, usesTessellationData: true); + var request = new ShaderCompileRequest(plan, resources, layout) + { LocalSizeX = 256, WaveSize = 64, CooperativeWave64Workgroup = true, TessellationHull = new(4, 4, 63, 0) }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + SharpEmu.ShaderCompiler.Tests.Gen5LargeDispatcherValidationTests.ValidateWithSpirvToolsWhenAvailable(shader.Spirv); + using var harness = new ImageTestHarness(vulkan); + using var runner = new LayoutComputeRunner(harness, request, shader.Spirv); + var buffer = runner.CreateBuffer(4096); + var scalar = new uint[64]; scalar[26] = 4096; + var data = new uint[Gen5TessellationData.DwordCount]; + data[Gen5TessellationData.FirstPatch] = 7; + data[Gen5TessellationData.PatchCount] = 63; + data[Gen5TessellationData.VertexOffset] = 11; + data[Gen5TessellationData.IndexSize] = indexSize; + data[Gen5TessellationData.IndexByteOffset] = byteBias; + uint Index(uint index) => indexSize switch { 1 => index % 127, 2 => 1000 + index, _ => 0xFEDC0000 + index }; + if (indexSize != 0) + { + var indices = new byte[(int)((284 * indexSize + byteBias + 3) & ~3u)]; + for (uint index = 0; index < 284; index++) + { + var offset = (int)(byteBias + index * indexSize); + if (indexSize == 1) indices[offset] = (byte)Index(index); + else if (indexSize == 2) BinaryPrimitives.WriteUInt16LittleEndian(indices.AsSpan(offset), (ushort)Index(index)); + else BinaryPrimitives.WriteUInt32LittleEndian(indices.AsSpan(offset), Index(index)); + } + var indexBuffer = runner.CreateBuffer(indices); + data[Gen5TessellationData.IndexAddress] = (uint)indexBuffer.DeviceAddress; + data[Gen5TessellationData.IndexAddress + 1] = (uint)(indexBuffer.DeviceAddress >> 32); + } + harness.Run(() => runner.Dispatch(scalar, + new Dictionary { [DescriptorBindingKind.Buffers] = [buffer] }, 1, + tessellationData: data)); + var bytes = runner.ReadBack(buffer, 0, 4096); + for (uint lane = 0; lane < 256; lane++) + { + uint Read(int component) => BinaryPrimitives.ReadUInt32LittleEndian(bytes.AsSpan((int)lane * 16 + component * 4)); + var count = lane < 192 ? 64u : 60u; + Assert.Equal(count | (count << 8), Read(0)); + Assert.Equal(7 + lane / 4, Read(1)); + Assert.Equal((lane / 4) | ((lane % 4) << 8), Read(2)); + var sourceIndex = 7 * 4 + lane; + Assert.Equal((indexSize != 0 && lane < 252 ? Index(sourceIndex) : sourceIndex) + 11, Read(3)); + } + harness.AssertNoValidationMessages(); + } +} diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/Wave64HalfMaskDeviceTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/Wave64HalfMaskDeviceTests.cs index fb9f391..eb4d94b 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/Wave64HalfMaskDeviceTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/Wave64HalfMaskDeviceTests.cs @@ -206,6 +206,35 @@ public sealed class Wave64HalfMaskDeviceTests(HeadlessVulkanFixture fixture) : I } } + // A march loop shaped like Ghost of Yotei's lighting pass: a scalar step counter compared + // into EXEC, lanes leaving through VCC as well, and the back edge taken while EXEC is not + // empty. Each lane must stop on its own, in both halves of the wave. + [Fact] + public void AMarchLoop_EndsEveryLaneByItsCounterOrItsCompare() + { + if (!Ready()) return; + Lane(); + Add(Vop2(0, "VAndB32", 3, Operand(15), Gen5Operand.Vector(2))); + Add(MoveVector(0, 4, 0)); + Add(Sop1(0, "SMovB32", 2, Operand(0))); + var header = Add(Vopc(0, "VCmpxLtI32", Gen5Operand.Scalar(2), 3)); + var exit = Add(Branch(0, "SCbranchExecz", 0)); + Add(Vop2(0, "VAddU32", 4, Operand(1), Gen5Operand.Vector(4))); + Add(Sop2(0, "SAddI32", 2, Gen5Operand.Scalar(2), Operand(1))); + Add(Vopc(0, "VCmpLeU32", Operand(5), 4)); + Add(Sop2(0, "SAndn2B64", Exec, Gen5Operand.Scalar(Exec), Gen5Operand.Scalar(Vcc))); + var back = Add(Branch(0, "SCbranchExecnz", 0)); + var done = Add(Sop1(0, "SMovB64", Exec, AllLanes)); + Point(exit, done); + Point(back, header); + + var lanes = Run(); + for (uint lane = 0; lane < 64; lane++) + { + Assert.Equal(Math.Min(lane & 15, 5u), lanes[lane]); + } + } + [Theory] [InlineData(false)] [InlineData(true)] @@ -281,6 +310,38 @@ public sealed class Wave64HalfMaskDeviceTests(HeadlessVulkanFixture fixture) : I harness.AssertNoValidationMessages(); } + // Ghost of Yotei's non-uniform texture loop: READFIRSTLANE into VCC_LO, an SDWA compare + // that writes its lane mask to an SGPR pair, AND_SAVEEXEC, then EXEC = saved & ~mask until + // EXEC is empty. Every lane must be visited exactly once. + [Fact] + public void AWaterfallLoop_WithAnSdwaCompareToAnSgprPair_VisitsEachLaneOnce() + { + if (!Ready()) return; + // v_cmp_eq_u32_sdwa s[98:99], s106, v3 (from the game's code, with v3 for v162). + uint[] words = [0x7D8406F9u, 0x0686E26Au, 0xBF810000u]; + var bytes = new byte[words.Length * sizeof(uint)]; + for (var index = 0; index < words.Length; index++) + BinaryPrimitives.WriteUInt32LittleEndian(bytes.AsSpan(index * sizeof(uint)), words[index]); + Assert.True(Gen5ShaderTranslator.TryDecodeProgram(new SharpEmu.HLE.CpuContext(new WordMemory(bytes), SharpEmu.HLE.Generation.Gen5), + 0x1000, out var decoded, out var decodeError), decodeError); + var compare = decoded.Instructions[0]; + Assert.Equal("VCmpEqU32", compare.Opcode); + + Lane(); + Add(Vop2(0, "VAndB32", 3, Operand(3), Gen5Operand.Vector(2))); + Add(MoveVector(0, 4, 0)); + var header = Add(ReadFirstLane(0, Vcc, 3)); + Add(compare); + Add(Sop1(0, "SAndSaveexecB64", 100, Gen5Operand.Scalar(98))); + Add(Vop2(0, "VAddU32", 4, Operand(1), Gen5Operand.Vector(4))); + Add(Sop2(0, "SAndn2B64", Exec, Gen5Operand.Scalar(100), Gen5Operand.Scalar(98))); + var back = Add(Branch(0, "SCbranchExecnz", 0)); + Add(Sop1(0, "SMovB64", Exec, AllLanes)); + Point(back, header); + + Assert.All(Run(), count => Assert.Equal(1u, count)); + } + private sealed class WordMemory(byte[] bytes) : SharpEmu.HLE.ICpuMemory { public bool TryRead(ulong address, Span destination) diff --git a/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/WaveShuffleDeviceTests.cs b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/WaveShuffleDeviceTests.cs index bd09267..2ebadd9 100644 --- a/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/WaveShuffleDeviceTests.cs +++ b/tests/SharpEmu.Libs.Tests/Gpu/Vulkan/WaveShuffleDeviceTests.cs @@ -17,12 +17,16 @@ public sealed class WaveShuffleDeviceTests(HeadlessVulkanFixture fixture, ITestO : IClassFixture { [Theory] - [InlineData("Dpp8")] - [InlineData("Dpp16")] - [InlineData("Permlane16")] - [InlineData("Permlanex16")] - [InlineData("Reduction")] - public void Wave64Shuffle_PreservesUpperHalf(string operation) + [InlineData("Dpp8", 32)] + [InlineData("Dpp8", 64)] + [InlineData("Dpp16", 32)] + [InlineData("Dpp16", 64)] + [InlineData("Permlane16", 32)] + [InlineData("Permlane16", 64)] + [InlineData("Permlanex16", 32)] + [InlineData("Permlanex16", 64)] + [InlineData("Reduction", 64)] + public void WaveShuffle_PreservesEachHalf(string operation, uint waveSize) { var vulkan = fixture.Vulkan; if (!GatePrerequisites.Ready(vulkan, shaderInt64: true)) return; @@ -79,7 +83,7 @@ public sealed class WaveShuffleDeviceTests(HeadlessVulkanFixture fixture, ITestO var (plan, resources, layout) = Prepare(Program([.. instructions])); var request = new ShaderCompileRequest(plan, resources, layout) { - LocalSizeX = 8, LocalSizeY = 8, ThreadCountX = 8, ThreadCountY = 8, WaveSize = 64, + LocalSizeX = 8, LocalSizeY = waveSize / 8, ThreadCountX = 8, ThreadCountY = waveSize / 8, WaveSize = waveSize, }; Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); using var harness = new ImageTestHarness(vulkan); @@ -90,7 +94,7 @@ public sealed class WaveShuffleDeviceTests(HeadlessVulkanFixture fixture, ITestO harness.Run(() => runner.Dispatch(registers, new Dictionary { [DescriptorBindingKind.Buffers] = [result] }, 1)); var actual = runner.ReadBack(result, 0, 256); - for (uint lane = 0; lane < 64; lane++) + for (uint lane = 0; lane < waveSize; lane++) { var expected = operation switch { @@ -104,6 +108,6 @@ public sealed class WaveShuffleDeviceTests(HeadlessVulkanFixture fixture, ITestO Assert.Equal(expected, BinaryPrimitives.ReadUInt32LittleEndian(actual.AsSpan((int)lane * 4))); } harness.AssertNoValidationMessages(); - output.WriteLine($"Verified {operation} on {vulkan.DeviceName}."); + output.WriteLine($"Verified wave{waveSize} {operation} on {vulkan.DeviceName}."); } } diff --git a/tests/SharpEmu.Libs.Tests/Kernel/KernelAioWaitTests.cs b/tests/SharpEmu.Libs.Tests/Kernel/KernelAioWaitTests.cs index e40a8b6..33c0353 100644 --- a/tests/SharpEmu.Libs.Tests/Kernel/KernelAioWaitTests.cs +++ b/tests/SharpEmu.Libs.Tests/Kernel/KernelAioWaitTests.cs @@ -16,6 +16,32 @@ public sealed class KernelAioWaitTests private const ulong IdAddress = MemoryBase + 0x200; private const ulong StateAddress = MemoryBase + 0x300; + [Theory] + [InlineData(Generation.Gen4, false, false)] + [InlineData(Generation.Gen5, false, false)] + [InlineData(Generation.Gen5, true, false)] + [InlineData(Generation.Gen5, false, true)] + public void PollSingleResolvesAndReportsRequestState(Generation generation, bool deleted, bool unmapped) + { + var manager = new ModuleManager(); + manager.RegisterExports(SharpEmu.Generated.SysAbiExportRegistry.CreateExports(generation)); + Assert.True(manager.TryGetExport("2pOuoWoCxdk", out var export)); + Assert.Equal("sceKernelAioPollRequest", export.Name); + Assert.Equal("libKernel", export.LibraryName); + var context = CreateSubmission(out var submitId); + try + { + if (deleted) Delete(context, submitId); + context[CpuRegister.Rdi] = submitId; + context[CpuRegister.Rsi] = unmapped ? 0 : StateAddress; + var expected = deleted ? OrbisGen2Result.ORBIS_GEN2_ERROR_INVALID_ARGUMENT + : unmapped ? OrbisGen2Result.ORBIS_GEN2_ERROR_MEMORY_FAULT : OrbisGen2Result.ORBIS_GEN2_OK; + Assert.Equal((int)expected, export.Function(context)); + Assert.Equal(deleted || unmapped ? 0u : 3u, ReadState(context, StateAddress)); + } + finally { Delete(context, submitId); } + } + [Theory] [InlineData(false)] [InlineData(true)] diff --git a/tests/SharpEmu.Libs.Tests/Kernel/KernelEventQueueWaiterLifetimeTests.cs b/tests/SharpEmu.Libs.Tests/Kernel/KernelEventQueueWaiterLifetimeTests.cs index 19f2546..00f8d6b 100644 --- a/tests/SharpEmu.Libs.Tests/Kernel/KernelEventQueueWaiterLifetimeTests.cs +++ b/tests/SharpEmu.Libs.Tests/Kernel/KernelEventQueueWaiterLifetimeTests.cs @@ -15,6 +15,58 @@ public sealed class KernelEventQueueWaiterLifetimeTests private const ulong EventsAddress = BaseAddress + 0x200; private const ulong OutCountAddress = BaseAddress + 0x300; + [Theory] + [InlineData(false)] + [InlineData(true)] + public void EventFlag_CompletedWaitDoesNotRemainInCancelCount(bool signal) + { + var memory = new FakeCpuMemory(BaseAddress, 0x1000); + var ctx = new CpuContext(memory, Generation.Gen5); + Assert.True(memory.TryWrite(EventsAddress, "wait-lifetime\0"u8)); + ctx[CpuRegister.Rdi] = HandleAddress; + ctx[CpuRegister.Rsi] = EventsAddress; + ctx[CpuRegister.Rdx] = 0x20; + Assert.Equal(0, KernelEventFlagCompatExports.KernelCreateEventFlag(ctx)); + var handle = ReadUInt64(memory, HandleAddress); + for (var attempt = 0; attempt < 3; attempt++) + { + var previousThread = GuestThreadExecution.EnterGuestThread(0x703); + var previousFrame = GuestThreadExecution.EnterImportCallFrame(0x10000, 0x20000, 0x30000); + IGuestThreadBlockWaiter waiter; + try + { + ctx[CpuRegister.Rdi] = handle; + ctx[CpuRegister.Rsi] = 1; + ctx[CpuRegister.Rdx] = 0x12; + ctx[CpuRegister.Rcx] = 0; + ctx[CpuRegister.R8] = OutCountAddress; + Assert.Equal(0, KernelEventFlagCompatExports.KernelWaitEventFlag(ctx)); + Assert.True(GuestThreadExecution.TryConsumeCurrentThreadBlock( + out _, out _, out _, out _, out var staged, out _)); + waiter = Assert.IsAssignableFrom(staged); + } + finally + { + GuestThreadExecution.RestoreImportCallFrame(previousFrame); + GuestThreadExecution.RestoreGuestThread(previousThread); + } + if (signal) + { + ctx[CpuRegister.Rdi] = handle; + ctx[CpuRegister.Rsi] = 1; + Assert.Equal(0, KernelEventFlagCompatExports.KernelSetEventFlag(ctx)); + Assert.True(waiter.TryWake()); + } + Assert.Equal(signal ? 0 : (int)OrbisGen2Result.ORBIS_GEN2_ERROR_TIMED_OUT, waiter.Resume()); + } + ctx[CpuRegister.Rdi] = handle; + ctx[CpuRegister.Rsi] = 0; + ctx[CpuRegister.Rdx] = OutCountAddress; + Assert.Equal(0, KernelEventFlagCompatExports.KernelCancelEventFlag(ctx)); + Assert.Equal(0u, ReadUInt32(memory, OutCountAddress)); + ctx[CpuRegister.Rdi] = handle; + Assert.Equal(0, KernelEventFlagCompatExports.KernelDeleteEventFlag(ctx)); + } [Fact] public void DeleteEqueue_CompletesStagedWaiterAsDeleted() { diff --git a/tests/SharpEmu.Libs.Tests/Kernel/KernelMemoryCompatExportsTests.cs b/tests/SharpEmu.Libs.Tests/Kernel/KernelMemoryCompatExportsTests.cs index 6ca7bbe..7691e4b 100644 --- a/tests/SharpEmu.Libs.Tests/Kernel/KernelMemoryCompatExportsTests.cs +++ b/tests/SharpEmu.Libs.Tests/Kernel/KernelMemoryCompatExportsTests.cs @@ -27,6 +27,70 @@ public sealed class KernelMemoryCompatExportsTests private const ulong SpanStartOutAddress = GuestMemoryBase + 0x108; private const ulong SpanSizeOutAddress = GuestMemoryBase + 0x110; + [Theory] + [InlineData("/dev/urandom")] + [InlineData("/dev/random")] + public void EntropyDevice_ReadsFreshBytesAndCloses(string path) + { + var memory = new FakeCpuMemory(GuestMemoryBase, 0x2000); + var ctx = new CpuContext(memory, Generation.Gen5); + Assert.True(memory.TryWrite(GuestMemoryBase, Encoding.UTF8.GetBytes(path + "\0"))); + ctx[CpuRegister.Rdi] = GuestMemoryBase; + ctx[CpuRegister.Rsi] = 0; + Assert.Equal(0, KernelMemoryCompatExports.PosixOpen(ctx)); + var fd = ctx[CpuRegister.Rax]; + Assert.True(fd > 2); + try + { + ctx[CpuRegister.Rdi] = fd; + ctx[CpuRegister.Rsi] = GuestMemoryBase + 0x200; + Assert.Equal(0, KernelMemoryCompatExports.PosixFstat(ctx)); + var stat = new byte[120]; + Assert.True(memory.TryRead(GuestMemoryBase + 0x200, stat)); + Assert.Equal(0x2000, BitConverter.ToUInt16(stat, 8) & 0xF000); + Assert.Equal(0L, BitConverter.ToInt64(stat, 72)); + var first = new byte[64]; + var second = new byte[64]; + ctx[CpuRegister.Rdi] = fd; + ctx[CpuRegister.Rsi] = GuestMemoryBase + 0x100; + ctx[CpuRegister.Rdx] = 64; + Assert.Equal(0, KernelMemoryCompatExports.PosixRead(ctx)); + Assert.Equal(64UL, ctx[CpuRegister.Rax]); + Assert.True(memory.TryRead(GuestMemoryBase + 0x100, first)); + Assert.Equal(0, KernelMemoryCompatExports.PosixRead(ctx)); + Assert.True(memory.TryRead(GuestMemoryBase + 0x100, second)); + Assert.False(first.SequenceEqual(second)); + Assert.Contains(first, value => value != 0); + ctx[CpuRegister.Rsi] = 0; + Assert.Equal((int)OrbisGen2Result.ORBIS_GEN2_ERROR_INVALID_ARGUMENT, + KernelMemoryCompatExports.KernelReadUnderscore(ctx)); + } + finally + { + ctx[CpuRegister.Rdi] = fd; + Assert.Equal(0, KernelMemoryCompatExports.PosixClose(ctx)); + } + ctx[CpuRegister.Rsi] = GuestMemoryBase + 0x100; + ctx[CpuRegister.Rdx] = 64; + Assert.Equal(-1, KernelMemoryCompatExports.PosixRead(ctx)); + Assert.Equal(-1, KernelMemoryCompatExports.PosixFstat(ctx)); + Assert.Equal(-1, KernelMemoryCompatExports.PosixClose(ctx)); + } + + [Theory] + [InlineData(1)] + [InlineData(2)] + [InlineData(0x20000)] + public void EntropyDevice_RejectsUnsupportedAccess(int flags) + { + var memory = new FakeCpuMemory(GuestMemoryBase, 0x1000); + var ctx = new CpuContext(memory, Generation.Gen5); + Assert.True(memory.TryWrite(GuestMemoryBase, Encoding.UTF8.GetBytes("/dev/urandom\0"))); + ctx[CpuRegister.Rdi] = GuestMemoryBase; + ctx[CpuRegister.Rsi] = unchecked((ulong)flags); + Assert.Equal(-1, KernelMemoryCompatExports.PosixOpen(ctx)); + } + [Theory] [InlineData(false)] [InlineData(true)] diff --git a/tests/SharpEmu.Libs.Tests/Loader/GuestRedZonePatcherTests.cs b/tests/SharpEmu.Libs.Tests/Loader/GuestRedZonePatcherTests.cs index 82f7105..648b58a 100644 --- a/tests/SharpEmu.Libs.Tests/Loader/GuestRedZonePatcherTests.cs +++ b/tests/SharpEmu.Libs.Tests/Loader/GuestRedZonePatcherTests.cs @@ -4,6 +4,7 @@ using System.Buffers.Binary; using System.Runtime.InteropServices; using Iced.Intel; +using static Iced.Intel.AssemblerRegisters; using SharpEmu.Core.Loader; using SharpEmu.Core.Memory; using SharpEmu.HLE; @@ -258,6 +259,359 @@ public sealed class GuestRedZonePatcherTests Assert.Equal(6u, image.ReadUInt32(0x2000)); } + // Ghost of Tsushima 0x80067A7B9: a 16-bit compare, movzx after the bound and a table + // register defined before the loop that contains the dispatch. + [Fact] + public void ResolvesNarrowCompareWithZeroExtensionAndTableDefinedBeforeTheLoop() + { + var function = new JumpTableFunction(); + var a = function.Assembler; + var cases = function.Cases(6); + var table = function.Table(cases); + var loop = a.CreateLabel(); + var next = a.CreateLabel(); + a.lea(r15, __[table]); + a.Label(ref loop); + a.mov(edx, __dword_ptr[rdi]); + a.add(edx, r11d); + a.cmp(dx, 5); + a.ja(next); + a.movzx(edx, dx); + a.movsxd(rdx, __dword_ptr[r15 + rdx * 4]); + a.add(rdx, r15); + a.jmp(rdx); + a.Label(ref next); + a.add(rdi, 4); + a.jmp(loop); + function.EmitCases(cases, loop); + + function.AssertResolves(cases); + } + + // Ghost of Tsushima 0x80067A824: a load from the stack between the table lea and the load. + [Fact] + public void ResolvesTableWithUnrelatedInstructionsBeforeTheLoad() + { + var function = new JumpTableFunction(); + var a = function.Assembler; + var cases = function.Cases(4); + var table = function.Table(cases); + var done = a.CreateLabel(); + a.mov(r9d, __dword_ptr[rdi]); + a.dec(r9d); + a.cmp(r9d, 3); + a.ja(done); + a.lea(rax, __[table]); + a.mov(rcx, __qword_ptr[rsp - 0x68]); + a.movsxd(rdx, __dword_ptr[rax + r9 * 4]); + a.add(rdx, rax); + a.jmp(rdx); + a.Label(ref done); + a.ret(); + function.EmitCases(cases, done); + + function.AssertResolves(cases); + } + + // Ghost of Tsushima 0x80067A988 and 0x80067AA6D: the first dispatch redefines its table + // register after the add, the second reuses that register from every path into it. + [Fact] + public void ResolvesTableRegisterDefinedOnEveryPathIncludingJumpTableEdges() + { + var function = new JumpTableFunction(); + var a = function.Assembler; + var first = function.Cases(4); + var second = function.Cases(6); + var table = function.Table(first); + var secondTable = function.Table(second); + var outOfRange = a.CreateLabel(); + var join = a.CreateLabel(); + var done = a.CreateLabel(); + a.mov(r9d, __dword_ptr[rdi]); + a.cmp(r9d, 3); + a.ja(outOfRange); + a.lea(rcx, __[table]); + a.lea(r11, __[rbx + rdx]); + a.movsxd(rax, __dword_ptr[rcx + r9 * 4]); + a.add(rax, rcx); + a.lea(rcx, __[secondTable]); + a.jmp(rax); + a.Label(ref outOfRange); + a.lea(rcx, __[secondTable]); + a.jmp(join); + function.EmitCases(first, join); + a.Label(ref join); + a.mov(edx, __dword_ptr[rsi]); + a.lea(eax, __[rdx + 0x1000]); + a.cmp(ax, 5); + a.ja(done); + a.movzx(eax, ax); + a.movsxd(rax, __dword_ptr[rcx + rax * 4]); + a.add(rax, rcx); + a.jmp(rax); + a.Label(ref done); + a.ret(); + function.EmitCases(second, done); + + function.AssertResolves([.. first, .. second]); + } + + // Ghost of Tsushima 0x80067AB85: the selector is copied through a narrower register and + // the table register is reused for a byte store before its lea. + [Fact] + public void ResolvesSelectorThroughZeroExtensionWithTableRegisterReused() + { + var function = new JumpTableFunction(); + var a = function.Assembler; + var cases = function.Cases(0x1E); + var table = function.Table(cases); + var done = a.CreateLabel(); + a.mov(edx, __dword_ptr[rdi]); + a.lea(ebx, __[rdx - 0xE31]); + a.cmp(bx, 0x1D); + a.ja(done); + a.mov(al, 1); + a.movzx(ebx, bx); + a.mov(__dword_ptr[rsp - 0x78], eax); + a.lea(rax, __[table]); + a.movsxd(rbx, __dword_ptr[rax + rbx * 4]); + a.add(rbx, rax); + a.jmp(rbx); + a.Label(ref done); + a.ret(); + function.EmitCases(cases, done); + + function.AssertResolves(cases); + } + + [Fact] + public void RefusesUnboundedSelector() + { + var function = new JumpTableFunction(); + var a = function.Assembler; + var cases = function.Cases(2); + var table = function.Table(cases); + var done = a.CreateLabel(); + a.lea(r15, __[table]); + a.mov(edx, __dword_ptr[rdi]); + a.movsxd(rax, __dword_ptr[r15 + rdx * 4]); + a.add(rax, r15); + a.jmp(rax); + a.Label(ref done); + a.ret(); + function.EmitCases(cases, done); + + function.AssertRefuses(); + } + + [Fact] + public void RefusesNarrowCompareWithoutZeroExtension() + { + var function = new JumpTableFunction(); + var a = function.Assembler; + var cases = function.Cases(6); + var table = function.Table(cases); + var done = a.CreateLabel(); + a.lea(r15, __[table]); + a.mov(rdx, __qword_ptr[rdi]); + a.cmp(dx, 5); + a.ja(done); + a.movsxd(rax, __dword_ptr[r15 + rdx * 4]); + a.add(rax, r15); + a.jmp(rax); + a.Label(ref done); + a.ret(); + function.EmitCases(cases, done); + + function.AssertRefuses(); + } + + [Fact] + public void RefusesCompareOfAnotherRegister() + { + var function = new JumpTableFunction(); + var a = function.Assembler; + var cases = function.Cases(4); + var table = function.Table(cases); + var done = a.CreateLabel(); + a.mov(r9d, __dword_ptr[rdi]); + a.cmp(r8d, 3); + a.ja(done); + a.lea(rcx, __[table]); + a.movsxd(rax, __dword_ptr[rcx + r9 * 4]); + a.add(rax, rcx); + a.jmp(rax); + a.Label(ref done); + a.ret(); + function.EmitCases(cases, done); + + function.AssertRefuses(); + } + + [Fact] + public void RefusesSelectorRedefinedAfterTheBound() + { + var function = new JumpTableFunction(); + var a = function.Assembler; + var cases = function.Cases(4); + var table = function.Table(cases); + var done = a.CreateLabel(); + a.mov(r9d, __dword_ptr[rdi]); + a.cmp(r9d, 3); + a.ja(done); + a.inc(r9d); + a.lea(rcx, __[table]); + a.movsxd(rax, __dword_ptr[rcx + r9 * 4]); + a.add(rax, rcx); + a.jmp(rax); + a.Label(ref done); + a.ret(); + function.EmitCases(cases, done); + + function.AssertRefuses(); + } + + [Fact] + public void RefusesTwoDifferentTablesInTheSameRegister() + { + var function = new JumpTableFunction(); + var a = function.Assembler; + var cases = function.Cases(2); + var table = function.Table(cases); + var otherTable = function.Table(cases); + var other = a.CreateLabel(); + var join = a.CreateLabel(); + var done = a.CreateLabel(); + a.mov(r9d, __dword_ptr[rdi]); + a.test(esi, esi); + a.je(other); + a.lea(rcx, __[table]); + a.jmp(join); + a.Label(ref other); + a.lea(rcx, __[otherTable]); + a.Label(ref join); + a.cmp(r9d, 1); + a.ja(done); + a.movsxd(rax, __dword_ptr[rcx + r9 * 4]); + a.add(rax, rcx); + a.jmp(rax); + a.Label(ref done); + a.ret(); + function.EmitCases(cases, done); + + function.AssertRefuses(); + } + + [Fact] + public void RefusesTableRegisterLoadedFromMemoryOnOnePath() + { + var function = new JumpTableFunction(); + var a = function.Assembler; + var cases = function.Cases(2); + var table = function.Table(cases); + var other = a.CreateLabel(); + var join = a.CreateLabel(); + var done = a.CreateLabel(); + a.mov(r9d, __dword_ptr[rdi]); + a.test(esi, esi); + a.je(other); + a.lea(rcx, __[table]); + a.jmp(join); + a.Label(ref other); + a.mov(rcx, __qword_ptr[rdx]); + a.Label(ref join); + a.cmp(r9d, 1); + a.ja(done); + a.movsxd(rax, __dword_ptr[rcx + r9 * 4]); + a.add(rax, rcx); + a.jmp(rax); + a.Label(ref done); + a.ret(); + function.EmitCases(cases, done); + + function.AssertRefuses(); + } + + private sealed class JumpTableFunction + { + private const ulong BaseAddress = 0x8_0000_0000; + private readonly List<(Label Table, Label[] Entries)> _tables = []; + + public Assembler Assembler { get; } = new(64); + + public Label[] Cases(int count) + { + var cases = new Label[count]; + for (var index = 0; index < count; index++) + { + cases[index] = Assembler.CreateLabel(); + } + + return cases; + } + + public Label Table(Label[] entries) + { + var table = Assembler.CreateLabel(); + _tables.Add((table, entries)); + return table; + } + + public void EmitCases(Label[] cases, Label exit) + { + for (var index = 0; index < cases.Length; index++) + { + Assembler.Label(ref cases[index]); + Assembler.mov(eax, index); + Assembler.jmp(exit); + } + } + + public void AssertResolves(Label[] cases) + { + var (image, codeLength, result) = Assemble(); + Assert.True(GuestRedZonePatcher.TryResolveJumpTableTargets(image, BaseAddress, codeLength, out var targets)); + Assert.Equal(cases.Select(label => result.GetLabelRIP(label)).Distinct().Order(), targets); + } + + public void AssertRefuses() + { + var (image, codeLength, _) = Assemble(); + Assert.False(GuestRedZonePatcher.TryResolveJumpTableTargets(image, BaseAddress, codeLength, out _)); + } + + private (byte[] Image, int CodeLength, AssemblerResult Result) Assemble() + { + var codeEnd = Assembler.CreateLabel(); + Assembler.Label(ref codeEnd); + Assembler.int3(); + for (var index = 0; index < _tables.Count; index++) + { + var (table, entries) = _tables[index]; + Assembler.Label(ref table); + Assembler.db(new byte[entries.Length * sizeof(int)]); + _tables[index] = (table, entries); + } + + using var stream = new MemoryStream(); + var result = Assembler.Assemble( + new StreamCodeWriter(stream), BaseAddress, BlockEncoderOptions.ReturnNewInstructionOffsets); + var image = stream.ToArray(); + foreach (var (table, entries) in _tables) + { + var tableAddress = result.GetLabelRIP(table); + for (var index = 0; index < entries.Length; index++) + { + BinaryPrimitives.WriteInt32LittleEndian( + image.AsSpan((int)(tableAddress - BaseAddress) + index * sizeof(int)), + (int)(result.GetLabelRIP(entries[index]) - tableAddress)); + } + } + + return (image, (int)(result.GetLabelRIP(codeEnd) - BaseAddress), result); + } + } + private sealed unsafe class PatchedImage : IDisposable { private const ulong ImageSize = 0x10000; diff --git a/tests/SharpEmu.Libs.Tests/Memory/GpuMemory/GuestGpuMemoryTests.cs b/tests/SharpEmu.Libs.Tests/Memory/GpuMemory/GuestGpuMemoryTests.cs index 7504dfe..e9db06a 100644 --- a/tests/SharpEmu.Libs.Tests/Memory/GpuMemory/GuestGpuMemoryTests.cs +++ b/tests/SharpEmu.Libs.Tests/Memory/GpuMemory/GuestGpuMemoryTests.cs @@ -299,6 +299,20 @@ public sealed class GuestGpuMemoryTests _memory.Dispose(); } + [Fact] + public void MappingIntoUnmappedAddressesSkipsTheGpuDrain() + { + var scheduler = new RecordingScheduler { Active = true }; + var queue = new InlineQueue { IsGpuQueueThread = false }; + _memory.AttachGpuQueue(queue, scheduler); + _memory.RunMappingChange(() => scheduler.Calls.Add("change"), needsDrain: () => false); + // Still ordered after the queued GPU work, but no wait for the GPU to finish it. + Assert.Equal(new[] { "change" }, scheduler.Calls); + Assert.Equal(1, queue.Runs); + _memory.AttachGpuQueue(null, null); + _memory.Dispose(); + } + [Fact] public async Task MappingChangeWaitsForDetachWhenTheRelayCloses() { diff --git a/tests/SharpEmu.Libs.Tests/Memory/GpuMemory/GuestPageTrackerTests.cs b/tests/SharpEmu.Libs.Tests/Memory/GpuMemory/GuestPageTrackerTests.cs index 58a8fec..8a97835 100644 --- a/tests/SharpEmu.Libs.Tests/Memory/GpuMemory/GuestPageTrackerTests.cs +++ b/tests/SharpEmu.Libs.Tests/Memory/GpuMemory/GuestPageTrackerTests.cs @@ -153,6 +153,19 @@ public sealed class GuestPageTrackerTests : IDisposable Release(address, Page * 2); } + [NativePageProtectionFact] + public void RepeatedCpuDirtyMarkDoesNotRepeatTheOwnershipTransition() + { + var address = Allocate(1); + _tracker.ForEachUploadRange(address, Page, false, NoRange, NoUpload, preserveCpuWriteHotPages: false); + + Assert.True(_tracker.MarkCpuDirtyPages(address + 16, 8)); + Assert.False(_tracker.MarkCpuDirtyPages(address + 16, 8)); + + _tracker.UntrackMemory(address, Page); + Release(address, Page); + } + [NativePageProtectionFact] public void LockFreeCpuDirtyQueryAgreesWithTheLockedQuery() { @@ -627,6 +640,32 @@ public sealed class GuestPageTrackerTests : IDisposable Release(address, Page); } + [NativePageProtectionFact] + public void FrameDecayReturnsHotPagesToWriteTracking() + { + var address = Allocate(1); + _tracker.ForEachUploadRange(address, Page, false, NoRange, NoUpload); + _tracker.MarkCpuDirtyPages(address, Page); + _tracker.ForEachUploadRange(address, Page, false, NoRange, NoUpload); + _tracker.MarkCpuDirtyPages(address, Page); + Assert.True(_tracker.IsCpuWriteHotRange(address, Page)); + + _tracker.DecayCpuWriteHeat(); + Assert.False(_tracker.IsCpuWriteHotRange(address, Page)); + + // The pending CPU bytes are copied once more; then the page is clean and protected. + var uploads = 0; + _tracker.ForEachUploadRange(address, Page, false, (_, _) => uploads++, NoUpload); + Assert.Equal(1, uploads); + Assert.False(_tracker.HasCpuDirtyPages(address, Page)); + Assert.False(IsWritable(address)); + _tracker.ForEachUploadRange(address, Page, false, (_, _) => uploads++, NoUpload); + Assert.Equal(1, uploads); + + _tracker.UntrackMemory(address, Page); + Release(address, Page); + } + [NativePageProtectionFact] public void RepeatedCpuWritesKeepReadOnlyHotPagesWritableAndDirty() { @@ -739,6 +778,31 @@ public sealed class GuestPageTrackerTests : IDisposable Release(address, Page); } + [NativePageProtectionFact] + public void NestedUploadOnAnotherTrackerKeepsTheOuterRollbackState() + { + using var other = new GuestPageTrackerTests(); + var address = Allocate(1); + var otherAddress = other.Allocate(1); + Assert.Throws(() => + _tracker.ForEachUploadRange(address, Page, false, NoRange, () => + { + other._tracker.ForEachUploadRange(otherAddress, Page, false, NoRange, NoUpload); + throw new InvalidOperationException("outer upload failed"); + })); + + Assert.True(_tracker.HasCpuDirtyPages(address, Page)); + Assert.True(IsWritable(address)); + Assert.False(other._tracker.HasCpuDirtyPages(otherAddress, Page)); + var uploaded = 0; + _tracker.ForEachUploadRange(address, Page, false, (_, _) => uploaded++, NoUpload); + Assert.Equal(1, uploaded); + _tracker.UntrackMemory(address, Page); + Release(address, Page); + other._tracker.UntrackMemory(otherAddress, Page); + other.Release(otherAddress, Page); + } + [NativePageProtectionTheory] [InlineData(false, true)] [InlineData(true, true)] diff --git a/tests/SharpEmu.Libs.Tests/Memory/HostMemoryQueryRunTests.cs b/tests/SharpEmu.Libs.Tests/Memory/HostMemoryQueryRunTests.cs new file mode 100644 index 0000000..12c0700 --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Memory/HostMemoryQueryRunTests.cs @@ -0,0 +1,53 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.HLE; +using Xunit; + +namespace SharpEmu.Libs.Tests.Memory; + +// A POSIX query reports the run of pages sharing the queried page's protection, ending at the +// next protection change, as Win32 VirtualQuery does. +public sealed unsafe class HostMemoryQueryRunTests +{ + [Fact] + public void QueryRunsEndAtTheNextProtectionChange() + { + if (OperatingSystem.IsWindows()) return; + var page = (ulong)Environment.SystemPageSize; + var memory = (byte*)HostMemory.Alloc(null, (nuint)(page * 16), HostMemory.MEM_RESERVE | HostMemory.MEM_COMMIT, HostMemory.PAGE_READWRITE); + Assert.True(memory != null); + var start = (ulong)memory; + try + { + Assert.True(HostMemory.Protect((void*)(start + 3 * page), (nuint)(3 * page), HostMemory.PAGE_READONLY, out _)); + Assert.True(HostMemory.Protect((void*)(start + 9 * page), (nuint)page, HostMemory.PAGE_NOACCESS, out _)); + Assert.True(HostMemory.Protect((void*)(start + 15 * page), (nuint)page, HostMemory.PAGE_READONLY, out _)); + + AssertRun(start, 0, 3, HostMemory.PAGE_READWRITE); + AssertRun(start, 3, 3, HostMemory.PAGE_READONLY); + AssertRun(start, 4, 2, HostMemory.PAGE_READONLY); + AssertRun(start, 6, 3, HostMemory.PAGE_READWRITE); + AssertRun(start, 9, 1, HostMemory.PAGE_NOACCESS); + AssertRun(start, 10, 5, HostMemory.PAGE_READWRITE); + // A run at the last page of the mapping. + AssertRun(start, 15, 1, HostMemory.PAGE_READONLY); + + // Restoring the default protection removes the overrides from the run. + Assert.True(HostMemory.Protect((void*)(start + 3 * page), (nuint)(7 * page), HostMemory.PAGE_READWRITE, out _)); + AssertRun(start, 0, 15, HostMemory.PAGE_READWRITE); + } + finally + { + Assert.True(HostMemory.Free(memory, 0, HostMemory.MEM_RELEASE)); + } + + void AssertRun(ulong baseAddress, ulong firstPage, ulong pages, uint protect) + { + Assert.NotEqual(0u, (uint)HostMemory.Query((void*)(baseAddress + firstPage * page + 8), out var info)); + Assert.Equal(baseAddress + firstPage * page, info.BaseAddress); + Assert.Equal(pages * page, info.RegionSize); + Assert.Equal(protect, info.Protect); + } + } +} diff --git a/tests/SharpEmu.Libs.Tests/Network/NetExportsTests.cs b/tests/SharpEmu.Libs.Tests/Network/NetExportsTests.cs index bdbfaf6..e700ea3 100644 --- a/tests/SharpEmu.Libs.Tests/Network/NetExportsTests.cs +++ b/tests/SharpEmu.Libs.Tests/Network/NetExportsTests.cs @@ -14,6 +14,272 @@ public sealed class NetExportsTests { private readonly CpuContext _ctx = new(new FakeCpuMemory(0x1_0000_0000, 0x1000), Generation.Gen5); + [Fact] + public void EthernetFormattingWritesExactlyEighteenBytesAndSupportsOverlap() + { + const ulong address = 0x1_0000_0100; + var initial = Enumerable.Repeat((byte)0xA5, 20).ToArray(); + new byte[] { 0x02, 0x7F, 0x80, 0xAB, 0xCD, 0xFF }.CopyTo(initial, 0); + Assert.True(_ctx.Memory.TryWrite(address, initial)); + _ctx[CpuRegister.Rdi] = address; + _ctx[CpuRegister.Rsi] = address; + _ctx[CpuRegister.Rdx] = 18; + Assert.Equal(0, NetExports.NetEtherNtostr(_ctx)); + var actual = new byte[20]; + Assert.True(_ctx.Memory.TryRead(address, actual)); + Assert.Equal(System.Text.Encoding.ASCII.GetBytes("02:7f:80:ab:cd:ff\0"), actual[..18]); + Assert.Equal(new byte[] { 0xA5, 0xA5 }, actual[18..]); + } + + [Theory] + [InlineData(17)] + [InlineData(-1)] + public void EthernetFormattingRejectsSmallOrNegativeCapacityWithoutWriting(int capacity) + { + const ulong address = 0x1_0000_0100; + var sentinel = Enumerable.Repeat((byte)0xA5, 18).ToArray(); + Assert.True(_ctx.Memory.TryWrite(address, sentinel)); + _ctx[CpuRegister.Rdi] = address; + _ctx[CpuRegister.Rsi] = address; + _ctx[CpuRegister.Rdx] = unchecked((ulong)capacity); + Assert.Equal(unchecked((int)0x80410116), NetExports.NetEtherNtostr(_ctx)); + var actual = new byte[18]; + Assert.True(_ctx.Memory.TryRead(address, actual)); + Assert.Equal(sentinel, actual); + } + + [Fact] + public void EpollReadiness_DistinguishesPendingAcceptFromPeerShutdown() + { + _ctx[CpuRegister.Rsi] = 2; + _ctx[CpuRegister.Rdx] = 1; + _ctx[CpuRegister.Rcx] = 6; + Assert.Equal(0, NetExports.NetSocket(_ctx)); + var socketId = _ctx[CpuRegister.Rax]; + var sockets = (System.Collections.Concurrent.ConcurrentDictionary) + typeof(NetExports).GetField("_sockets", System.Reflection.BindingFlags.Static | + System.Reflection.BindingFlags.NonPublic)!.GetValue(null)!; + var listener = sockets[(int)socketId]; + listener.Bind(new System.Net.IPEndPoint(System.Net.IPAddress.Loopback, 0)); + _ctx[CpuRegister.Rdi] = socketId; + _ctx[CpuRegister.Rsi] = 1; + Assert.Equal(0, NetExports.NetListen(_ctx)); + _ctx[CpuRegister.Rdi] = 0; + _ctx[CpuRegister.Rsi] = 0; + Assert.Equal(0, NetExports.NetEpollCreate(_ctx)); + var poll = _ctx[CpuRegister.Rax]; + ulong acceptedId = 0; + using var client = new System.Net.Sockets.Socket(System.Net.Sockets.AddressFamily.InterNetwork, + System.Net.Sockets.SocketType.Stream, System.Net.Sockets.ProtocolType.Tcp); + const ulong input = 0x1_0000_0100, output = 0x1_0000_0200; + try + { + var record = new byte[24]; + System.Buffers.Binary.BinaryPrimitives.WriteUInt32LittleEndian(record, 1); + Assert.True(_ctx.Memory.TryWrite(input, record)); + void Watch(ulong id) + { + _ctx[CpuRegister.Rdi] = poll; + _ctx[CpuRegister.Rsi] = 1; + _ctx[CpuRegister.Rdx] = id; + _ctx[CpuRegister.Rcx] = input; + Assert.Equal(0, NetExports.NetEpollControl(_ctx)); + } + uint ReadEvents() + { + _ctx[CpuRegister.Rdi] = poll; + _ctx[CpuRegister.Rsi] = output; + _ctx[CpuRegister.Rdx] = 1; + _ctx[CpuRegister.Rcx] = 1_000_000; + Assert.Equal(0, NetExports.NetEpollWait(_ctx)); + Assert.Equal(1UL, _ctx[CpuRegister.Rax]); + Assert.True(_ctx.Memory.TryRead(output, record)); + return System.Buffers.Binary.BinaryPrimitives.ReadUInt32LittleEndian(record); + } + Watch(socketId); + client.Connect(listener.LocalEndPoint!); + Assert.Equal(1u, ReadEvents()); + _ctx[CpuRegister.Rdi] = socketId; + Assert.Equal(0, NetExports.NetAccept(_ctx)); + acceptedId = _ctx[CpuRegister.Rax]; + Watch(acceptedId); + client.Shutdown(System.Net.Sockets.SocketShutdown.Send); + Assert.Equal(0x11u, ReadEvents()); + } + finally + { + if (acceptedId != 0) + { + _ctx[CpuRegister.Rdi] = acceptedId; + NetExports.NetSocketClose(_ctx); + } + _ctx[CpuRegister.Rdi] = socketId; + NetExports.NetSocketClose(_ctx); + _ctx[CpuRegister.Rdi] = poll; + NetExports.NetEpollDestroy(_ctx); + } + } + + [Fact] + public void EpollDestroy_WakesAnInfiniteWaitWithBadDescriptor() + { + Assert.Equal(0, NetExports.NetEpollCreate(_ctx)); + var poll = _ctx[CpuRegister.Rax]; + using var started = new ManualResetEventSlim(); + Exception? failure = null; + var result = 0; + var worker = new Thread(() => + { + try + { + var context = new CpuContext(_ctx.Memory, Generation.Gen5); + context[CpuRegister.Rdi] = poll; + context[CpuRegister.Rsi] = 0x1_0000_0100; + context[CpuRegister.Rdx] = 1; + context[CpuRegister.Rcx] = uint.MaxValue; + started.Set(); + result = NetExports.NetEpollWait(context); + } + catch (Exception error) { failure = error; } + }) { IsBackground = true }; + worker.Start(); + try + { + Assert.True(started.Wait(TimeSpan.FromSeconds(5))); + Assert.False(worker.Join(TimeSpan.FromMilliseconds(20))); + } + finally + { + _ctx[CpuRegister.Rdi] = poll; + Assert.Equal(0, NetExports.NetEpollDestroy(_ctx)); + Assert.True(worker.Join(TimeSpan.FromSeconds(5))); + } + Assert.Null(failure); + Assert.Equal(unchecked((int)0x80410109), result); + } + + [Fact] + public void EpollWait_EmptyPollWaitsForMicrosecondTimeoutWithoutWritingEvents() + { + Assert.Equal(0, NetExports.NetEpollCreate(_ctx)); + var poll = _ctx[CpuRegister.Rax]; + var output = 0x1_0000_0100UL; + var before = Enumerable.Repeat((byte)0xCC, 48).ToArray(); + Assert.True(_ctx.Memory.TryWrite(output, before)); + try + { + _ctx[CpuRegister.Rdi] = poll; + _ctx[CpuRegister.Rsi] = output; + _ctx[CpuRegister.Rdx] = 2; + _ctx[CpuRegister.Rcx] = 10_000; + var clock = System.Diagnostics.Stopwatch.StartNew(); + Assert.Equal(0, NetExports.NetEpollWait(_ctx)); + Assert.Equal(0UL, _ctx[CpuRegister.Rax]); + Assert.True(clock.Elapsed >= TimeSpan.FromMilliseconds(10)); + var after = new byte[48]; + Assert.True(_ctx.Memory.TryRead(output, after)); + Assert.Equal(before, after); + } + finally { _ctx[CpuRegister.Rdi] = poll; NetExports.NetEpollDestroy(_ctx); } + } + + [Fact] + public void EpollControlAndWait_ReportSocketReadinessAndPreserveEventData() + { + Assert.Equal(0, NetExports.NetEpollCreate(_ctx)); + var poll = _ctx[CpuRegister.Rax]; + _ctx[CpuRegister.Rsi] = 2; + _ctx[CpuRegister.Rdx] = 2; + _ctx[CpuRegister.Rcx] = 17; + Assert.Equal(0, NetExports.NetSocket(_ctx)); + var socket = _ctx[CpuRegister.Rax]; + const ulong input = 0x1_0000_0100, output = 0x1_0000_0200; + var record = new byte[24]; + System.Buffers.Binary.BinaryPrimitives.WriteUInt32LittleEndian(record, 2); + System.Buffers.Binary.BinaryPrimitives.WriteUInt64LittleEndian(record.AsSpan(16), 0x123456789ABCDEF0); + Assert.True(_ctx.Memory.TryWrite(input, record)); + Assert.True(_ctx.Memory.TryWrite(output, Enumerable.Repeat((byte)0xCC, 48).ToArray())); + try + { + _ctx[CpuRegister.Rdi] = poll; + _ctx[CpuRegister.Rsi] = 1; + _ctx[CpuRegister.Rdx] = socket; + _ctx[CpuRegister.Rcx] = input; + Assert.Equal(0, NetExports.NetEpollControl(_ctx)); + Assert.Equal(unchecked((int)0x80410111), NetExports.NetEpollControl(_ctx)); + _ctx[CpuRegister.Rsi] = output; + _ctx[CpuRegister.Rdx] = 1; + _ctx[CpuRegister.Rcx] = 0; + Assert.Equal(0, NetExports.NetEpollWait(_ctx)); + Assert.Equal(1UL, _ctx[CpuRegister.Rax]); + var actual = new byte[48]; + Assert.True(_ctx.Memory.TryRead(output, actual)); + Assert.Equal(2u, System.Buffers.Binary.BinaryPrimitives.ReadUInt32LittleEndian(actual)); + Assert.Equal(socket, System.Buffers.Binary.BinaryPrimitives.ReadUInt64LittleEndian(actual.AsSpan(8))); + Assert.Equal(0x123456789ABCDEF0UL, System.Buffers.Binary.BinaryPrimitives.ReadUInt64LittleEndian(actual.AsSpan(16))); + Assert.All(actual.Skip(24), value => Assert.Equal((byte)0xCC, value)); + System.Buffers.Binary.BinaryPrimitives.WriteUInt32LittleEndian(record, 1); + Assert.True(_ctx.Memory.TryWrite(input, record)); + _ctx[CpuRegister.Rsi] = 2; + _ctx[CpuRegister.Rdx] = socket; + _ctx[CpuRegister.Rcx] = input; + Assert.Equal(0, NetExports.NetEpollControl(_ctx)); + _ctx[CpuRegister.Rsi] = output; + _ctx[CpuRegister.Rdx] = 1; + _ctx[CpuRegister.Rcx] = 0; + Assert.Equal(0, NetExports.NetEpollWait(_ctx)); + Assert.Equal(0UL, _ctx[CpuRegister.Rax]); + _ctx[CpuRegister.Rsi] = 3; + _ctx[CpuRegister.Rdx] = socket; + _ctx[CpuRegister.Rcx] = 0; + Assert.Equal(0, NetExports.NetEpollControl(_ctx)); + Assert.Equal(unchecked((int)0x80410109), NetExports.NetEpollControl(_ctx)); + _ctx[CpuRegister.Rsi] = output; + _ctx[CpuRegister.Rdx] = 1; + Assert.Equal(0, NetExports.NetEpollWait(_ctx)); + Assert.Equal(0UL, _ctx[CpuRegister.Rax]); + } + finally + { + _ctx[CpuRegister.Rdi] = socket; NetExports.NetSocketClose(_ctx); + _ctx[CpuRegister.Rdi] = poll; NetExports.NetEpollDestroy(_ctx); + } + } + + [Fact] + public void EpollDescriptorsAreUniqueAndDestroyRejectsClosedDescriptors() + { + Assert.Equal(0, NetExports.NetEpollCreate(_ctx)); + var first = _ctx[CpuRegister.Rax]; + Assert.Equal(0, NetExports.NetEpollCreate(_ctx)); + var second = _ctx[CpuRegister.Rax]; + try + { + Assert.True(first > 0); + Assert.NotEqual(first, second); + _ctx[CpuRegister.Rdi] = first; + Assert.Equal(0, NetExports.NetEpollDestroy(_ctx)); + Assert.Equal(unchecked((int)0x80410109), NetExports.NetEpollDestroy(_ctx)); + } + finally + { + _ctx[CpuRegister.Rdi] = first; + NetExports.NetEpollDestroy(_ctx); + _ctx[CpuRegister.Rdi] = second; + NetExports.NetEpollDestroy(_ctx); + } + } + + [Fact] + public void EpollCreateRejectsFlagsAndUnreadableNames() + { + _ctx[CpuRegister.Rsi] = 1; + Assert.Equal(unchecked((int)0x80410116), NetExports.NetEpollCreate(_ctx)); + _ctx[CpuRegister.Rsi] = 0; + _ctx[CpuRegister.Rdi] = 1; + Assert.Equal(unchecked((int)0x8041010E), NetExports.NetEpollCreate(_ctx)); + } + [Fact] public void SocketDescriptorsFitGuestFdSet() { diff --git a/tests/SharpEmu.Libs.Tests/Pad/TouchContactIdsTests.cs b/tests/SharpEmu.Libs.Tests/Pad/TouchContactIdsTests.cs new file mode 100644 index 0000000..93d1cd4 --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/Pad/TouchContactIdsTests.cs @@ -0,0 +1,45 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.Libs.Pad; +using Xunit; + +namespace SharpEmu.Libs.Tests.Pad; + +public sealed class TouchContactIdsTests +{ + // Lifting the finger and touching again is a new stroke, so it needs a new id. + [Fact] + public void EachNewContactGetsANewId_AndKeepsItWhileDown() + { + var ids = new TouchContactIds(); + var first = ids.Track(0, down: true); + Assert.Equal(first, ids.Track(0, down: true)); + ids.Track(0, down: false); + var second = ids.Track(0, down: true); + + Assert.NotEqual(0, first); + Assert.NotEqual(first, second); + } + + [Fact] + public void TwoFingersGetDifferentIds() + { + var ids = new TouchContactIds(); + Assert.NotEqual(ids.Track(0, down: true), ids.Track(1, down: true)); + } + + [Fact] + public void IdsWrapFrom127BackTo1() + { + var ids = new TouchContactIds(); + byte last = 0; + for (var contact = 0; contact < 128; contact++) + { + last = ids.Track(0, down: true); + ids.Track(0, down: false); + } + + Assert.Equal(1, last); + } +} diff --git a/tests/SharpEmu.Libs.Tests/PlayGo/PlayGoExportsTests.cs b/tests/SharpEmu.Libs.Tests/PlayGo/PlayGoExportsTests.cs index 36b4896..5b3132c 100644 --- a/tests/SharpEmu.Libs.Tests/PlayGo/PlayGoExportsTests.cs +++ b/tests/SharpEmu.Libs.Tests/PlayGo/PlayGoExportsTests.cs @@ -26,6 +26,9 @@ public sealed class PlayGoExportsTests : IDisposable private const ulong HandleAddress = MemoryBase + 0x200; private const ulong ChunkIdsAddress = MemoryBase + 0x300; private const ulong LociAddress = MemoryBase + 0x400; + private const ulong OutEntriesAddress = MemoryBase + 0x500; + private const ulong ProgressAddress = MemoryBase + 0x600; + private const ulong NextChunkAddress = MemoryBase + 0x700; private readonly string? _originalApp0Root; private readonly string _app0Root; @@ -54,6 +57,79 @@ public sealed class PlayGoExportsTests : IDisposable Assert.Equal(new byte[] { LocusNotDownloaded }, ReadLoci(1)); } + [Fact] + public void GetLocus_MetadataFreeApp0_ReportsAllInstalledPakChunks() + { + for (var chunk = 0; chunk <= 8; chunk++) + { + File.WriteAllBytes(Path.Combine(_app0Root, $"pakchunk{chunk}-Windows.pak"), []); + } + + var handle = InitializeAndOpen(); + + Assert.Equal(BadChunkId, GetLocus(handle, [0, 8, 9])); + Assert.Equal( + new byte[] { LocusLocalFast, LocusLocalFast, LocusNotDownloaded }, + ReadLoci(3)); + } + + [Fact] + public void PlayGoPgm_ReportsEveryChunkAndProgress() + { + var cacheDirectory = Directory.CreateDirectory(Path.Combine(_app0Root, "cache_ps5")); + var header = new byte[0x14]; + "DMGP"u8.CopyTo(header); + BinaryPrimitives.WriteUInt32LittleEndian(header.AsSpan(0x10), 35); + File.WriteAllBytes(Path.Combine(cacheDirectory.FullName, "playgo.pgm"), header); + + var handle = InitializeAndOpen(); + + _ctx[CpuRegister.Rdi] = handle; + _ctx[CpuRegister.Rsi] = 0; + _ctx[CpuRegister.Rdx] = 0; + _ctx[CpuRegister.Rcx] = OutEntriesAddress; + Assert.Equal((int)OrbisGen2Result.ORBIS_GEN2_OK, PlayGoExports.PlayGoGetChunkId(_ctx)); + Assert.Equal(35u, ReadUInt32(OutEntriesAddress)); + + var chunkIds = Enumerable.Range(0, 35).Select(static value => (ushort)value).ToArray(); + WriteChunkIds(chunkIds); + _ctx[CpuRegister.Rsi] = ChunkIdsAddress; + _ctx[CpuRegister.Rdx] = (ulong)chunkIds.Length; + _ctx[CpuRegister.Rcx] = OutEntriesAddress; + Assert.Equal((int)OrbisGen2Result.ORBIS_GEN2_OK, PlayGoExports.PlayGoGetChunkId(_ctx)); + Assert.Equal(35u, ReadUInt32(OutEntriesAddress)); + Assert.Equal(chunkIds, ReadChunkIds(chunkIds.Length)); + + Assert.Equal((int)OrbisGen2Result.ORBIS_GEN2_OK, GetLocus(handle, [34])); + Assert.Equal(new byte[] { LocusLocalFast }, ReadLoci(1)); + + SetGetProgressArguments(handle, ChunkIdsAddress, (uint)chunkIds.Length, ProgressAddress); + Assert.Equal((int)OrbisGen2Result.ORBIS_GEN2_OK, PlayGoExports.PlayGoGetProgress(_ctx)); + Assert.Equal(1ul, ReadUInt64(ProgressAddress)); + Assert.Equal(1ul, ReadUInt64(ProgressAddress + sizeof(ulong))); + + _ctx[CpuRegister.Rdi] = handle; + _ctx[CpuRegister.Rsi] = NextChunkAddress; + Assert.Equal(unchecked((int)0x80020002), PlayGoExports.PlayGoRequestNextChunk(_ctx)); + Assert.Equal(0u, ReadUInt32(NextChunkAddress)); + } + + // playgo.pgm gives only a chunk count: the numbering 0..count-1 is assumed, so an ID + // outside it is not proof of a bad chunk. + [Fact] + public void PlayGoPgm_DoesNotRejectAnIdOutsideTheAssumedNumbering() + { + var cacheDirectory = Directory.CreateDirectory(Path.Combine(_app0Root, "cache_ps5")); + var header = new byte[0x14]; + "DMGP"u8.CopyTo(header); + BinaryPrimitives.WriteUInt32LittleEndian(header.AsSpan(0x10), 3); + File.WriteAllBytes(Path.Combine(cacheDirectory.FullName, "playgo.pgm"), header); + + var handle = InitializeAndOpen(); + + Assert.Equal((int)OrbisGen2Result.ORBIS_GEN2_OK, GetLocus(handle, [2, 40])); + } + [Fact] public void GetLocus_ParsedChunkDefinitions_WritesPrefixAndRejectsFirstUnknownChunk() { @@ -68,6 +144,7 @@ public sealed class PlayGoExportsTests : IDisposable [Theory] [InlineData(UnusableMetadataKind.DatOnly)] + [InlineData(UnusableMetadataKind.ScenarioOnly)] [InlineData(UnusableMetadataKind.MalformedChunkDefinitions)] [InlineData(UnusableMetadataKind.UnrecognizedChunkDefinitions)] public void GetLocus_UnparseableMetadata_RemainsPermissive(UnusableMetadataKind metadataKind) @@ -78,6 +155,10 @@ public sealed class PlayGoExportsTests : IDisposable var sceSys = Directory.CreateDirectory(Path.Combine(_app0Root, "sce_sys")); File.WriteAllBytes(Path.Combine(sceSys.FullName, "playgo-chunk.dat"), [0x70, 0x6C, 0x67, 0x6F]); break; + case UnusableMetadataKind.ScenarioOnly: + var scenarioDirectory = Directory.CreateDirectory(Path.Combine(_app0Root, "sce_sys")); + File.WriteAllText(Path.Combine(scenarioDirectory.FullName, "playgo-scenario.json"), "{}"); + break; case UnusableMetadataKind.MalformedChunkDefinitions: File.WriteAllText( Path.Combine(_app0Root, "playgo-chunkdefs.xml"), @@ -192,6 +273,33 @@ public sealed class PlayGoExportsTests : IDisposable return loci; } + private ushort[] ReadChunkIds(int count) + { + var bytes = new byte[count * sizeof(ushort)]; + Assert.True(_memory.TryRead(ChunkIdsAddress, bytes)); + var chunkIds = new ushort[count]; + for (var i = 0; i < count; i++) + { + chunkIds[i] = BinaryPrimitives.ReadUInt16LittleEndian(bytes.AsSpan(i * sizeof(ushort))); + } + + return chunkIds; + } + + private uint ReadUInt32(ulong address) + { + Span bytes = stackalloc byte[sizeof(uint)]; + Assert.True(_memory.TryRead(address, bytes)); + return BinaryPrimitives.ReadUInt32LittleEndian(bytes); + } + + private ulong ReadUInt64(ulong address) + { + Span bytes = stackalloc byte[sizeof(ulong)]; + Assert.True(_memory.TryRead(address, bytes)); + return BinaryPrimitives.ReadUInt64LittleEndian(bytes); + } + private void SetGetLocusArguments(uint handle, ulong chunkIds, uint count, ulong outLoci) { _ctx[CpuRegister.Rdi] = handle; @@ -200,9 +308,18 @@ public sealed class PlayGoExportsTests : IDisposable _ctx[CpuRegister.Rcx] = outLoci; } + private void SetGetProgressArguments(uint handle, ulong chunkIds, uint count, ulong outProgress) + { + _ctx[CpuRegister.Rdi] = handle; + _ctx[CpuRegister.Rsi] = chunkIds; + _ctx[CpuRegister.Rdx] = count; + _ctx[CpuRegister.Rcx] = outProgress; + } + public enum UnusableMetadataKind { DatOnly, + ScenarioOnly, MalformedChunkDefinitions, UnrecognizedChunkDefinitions, } diff --git a/tests/SharpEmu.Libs.Tests/Pthread/PthreadMutexSemanticsTests.cs b/tests/SharpEmu.Libs.Tests/Pthread/PthreadMutexSemanticsTests.cs index b27fa34..9790e7a 100644 --- a/tests/SharpEmu.Libs.Tests/Pthread/PthreadMutexSemanticsTests.cs +++ b/tests/SharpEmu.Libs.Tests/Pthread/PthreadMutexSemanticsTests.cs @@ -97,6 +97,73 @@ public sealed class PthreadMutexExceptionTests public sealed class PthreadMutexSemanticsTests { + [Fact] + public void TimedMutex_UncontendedZeroTimeoutAcquiresAndSelfWaitTimesOut() + { + const ulong address = 0x7_0000_0100; + var memory = new AllocatingCpuMemory(0x7_0000_0000, 0x4000); + var context = new CpuContext(memory, Generation.Gen5); + context[CpuRegister.Rdi] = address; + context[CpuRegister.Rsi] = 0; + Assert.Equal(0, KernelPthreadCompatExports.PthreadMutexTimedlock(context)); + try + { + context[CpuRegister.Rsi] = 10_000; + var elapsed = System.Diagnostics.Stopwatch.StartNew(); + Assert.Equal((int)OrbisGen2Result.ORBIS_GEN2_ERROR_TIMED_OUT, + KernelPthreadCompatExports.PthreadMutexTimedlock(context)); + Assert.True(elapsed.Elapsed >= TimeSpan.FromMilliseconds(10)); + } + finally + { + Assert.Equal(0, KernelPthreadCompatExports.PthreadMutexUnlock(context)); + } + Assert.Equal(0, KernelPthreadCompatExports.PthreadMutexTrylock(context)); + Assert.Equal(0, KernelPthreadCompatExports.PthreadMutexUnlock(context)); + Assert.Equal(0, KernelPthreadCompatExports.PthreadMutexDestroy(context)); + } + + [Fact] + public void TimedMutex_AcquiresAfterAnotherOwnerReleases() + { + const ulong address = 0x7_0001_0100; + var memory = new AllocatingCpuMemory(0x7_0001_0000, 0x4000); + var owner = new CpuContext(memory, Generation.Gen5); + owner[CpuRegister.Rdi] = address; + Assert.Equal(0, KernelPthreadCompatExports.PthreadMutexLock(owner)); + using var started = new ManualResetEventSlim(); + var result = -1; + Exception? failure = null; + var worker = new Thread(() => + { + try + { + var waiter = new CpuContext(memory, Generation.Gen5); + waiter[CpuRegister.Rdi] = address; + waiter[CpuRegister.Rsi] = 5_000_000; + started.Set(); + result = KernelPthreadCompatExports.PthreadMutexTimedlock(waiter); + if (result == 0) + Assert.Equal(0, KernelPthreadCompatExports.PthreadMutexUnlock(waiter)); + } + catch (Exception error) { failure = error; } + }) { IsBackground = true }; + worker.Start(); + try + { + Assert.True(started.Wait(TimeSpan.FromSeconds(5))); + Assert.False(worker.Join(TimeSpan.FromMilliseconds(20)), "The waiter acquired a held mutex."); + } + finally + { + Assert.Equal(0, KernelPthreadCompatExports.PthreadMutexUnlock(owner)); + Assert.True(worker.Join(TimeSpan.FromSeconds(10))); + Assert.Equal(0, KernelPthreadCompatExports.PthreadMutexDestroy(owner)); + } + Assert.Null(failure); + Assert.Equal(0, result); + } + [Fact] public void AdaptiveMutex_SelfLockIsIdempotent() { diff --git a/tests/SharpEmu.Libs.Tests/ShaderCompiler/Gen5ShaderTranslatorTests.cs b/tests/SharpEmu.Libs.Tests/ShaderCompiler/Gen5ShaderTranslatorTests.cs index 5854c7e..10cb279 100644 --- a/tests/SharpEmu.Libs.Tests/ShaderCompiler/Gen5ShaderTranslatorTests.cs +++ b/tests/SharpEmu.Libs.Tests/ShaderCompiler/Gen5ShaderTranslatorTests.cs @@ -235,15 +235,17 @@ public sealed class Gen5ShaderTranslatorTests Assert.Equal(0x1234u, instruction.Sources[1].Value); } - [Fact] - public void FusedProgramContinuesAfterSetProgramCounter() + [Theory] + [InlineData(0xBE802000u)] + [InlineData(0xBEFD2106u)] + public void FusedProgramContinuesAfterSetProgramCounter(uint tailJump) { const ulong continuationAddress = ProgramAddress + 0x100; const ulong entryHeaderAddress = ProgramAddress + 0x400; const ulong continuationHeaderAddress = ProgramAddress + 0x500; var memory = new FakeCpuMemory(ProgramAddress, 0x1000); - WriteWords(memory, ProgramAddress, 0xBF800000u, 0xBE802000u); + WriteWords(memory, ProgramAddress, 0xBF800000u, tailJump); WriteWords(memory, continuationAddress, 0xBF800000u, 0xBF810000u); WriteUInt32(memory, entryHeaderAddress + 0x44, 2 * sizeof(uint)); WriteUInt32(memory, continuationHeaderAddress + 0x44, 2 * sizeof(uint)); @@ -308,6 +310,27 @@ public sealed class Gen5ShaderTranslatorTests Assert.Equal(continuationAddress, unchecked(program.Address + program.Instructions[2].ProgramOffset)); } + [Fact] + public void EarlierFusedContinuation_PreservesTheGuestProgramCounter() + { + var memory = new FakeCpuMemory(ProgramAddress, 0x2000); + var entryAddress = ProgramAddress + 0x1000; + var continuationAddress = ProgramAddress + 0x100; + var entryHeader = ProgramAddress + 0x400; + var continuationHeader = ProgramAddress + 0x500; + WriteWords(memory, entryAddress, 0xBF800000u, 0xBEFD2106u); + WriteWords(memory, continuationAddress, 0xBE801F00u, 0xBF810000u); + WriteUInt32(memory, entryHeader + 0x44, 8); + WriteUInt32(memory, continuationHeader + 0x44, 8); + var context = new CpuContext(memory, Generation.Gen5); + Gen5ShaderTranslator.RegisterFusedProgram(context, entryAddress, entryHeader, continuationAddress, continuationHeader); + Assert.True(Gen5ShaderTranslator.TryDecodeProgram(context, entryAddress, out var program, out var error), error); + var getPc = Assert.Single(program.Instructions, instruction => instruction.Opcode == "SGetpcB64"); + Assert.Equal(0x100u, getPc.Pc); + Assert.Equal(continuationAddress + 4, + unchecked(program.Address + program.InstructionAddressOffset(getPc.Pc) + 4)); + } + private static void WriteWords(FakeCpuMemory memory, ulong address, params uint[] words) { var bytes = new byte[words.Length * sizeof(uint)]; diff --git a/tests/SharpEmu.Libs.Tests/SharpEmu.Libs.Tests.csproj b/tests/SharpEmu.Libs.Tests/SharpEmu.Libs.Tests.csproj index 6694cda..95d063b 100644 --- a/tests/SharpEmu.Libs.Tests/SharpEmu.Libs.Tests.csproj +++ b/tests/SharpEmu.Libs.Tests/SharpEmu.Libs.Tests.csproj @@ -24,4 +24,8 @@ SPDX-License-Identifier: GPL-2.0-or-later + + + + diff --git a/tests/SharpEmu.Libs.Tests/SystemService/SystemServiceExportsTests.cs b/tests/SharpEmu.Libs.Tests/SystemService/SystemServiceExportsTests.cs index f95a657..162e3fc 100644 --- a/tests/SharpEmu.Libs.Tests/SystemService/SystemServiceExportsTests.cs +++ b/tests/SharpEmu.Libs.Tests/SystemService/SystemServiceExportsTests.cs @@ -17,6 +17,22 @@ public sealed class SystemServiceExportsTests SystemServiceExports.ResetForTests(); } + [Fact] + public void EmptyEventPollPreservesTheEntireEventRecord() + { + var memory = new FakeCpuMemory(MemoryBase, 0x2004); + var sentinel = Enumerable.Repeat((byte)0xA5, 0x2004).ToArray(); + Assert.True(memory.TryWrite(MemoryBase, sentinel)); + var context = new CpuContext(memory, Generation.Gen5) { [CpuRegister.Rdi] = MemoryBase }; + for (var poll = 0; poll < 3; poll++) + Assert.Equal(unchecked((int)0x80A10004), SystemServiceExports.SystemServiceReceiveEvent(context)); + var actual = new byte[sentinel.Length]; + Assert.True(memory.TryRead(MemoryBase, actual)); + Assert.Equal(sentinel, actual); + context[CpuRegister.Rdi] = 0; + Assert.Equal(unchecked((int)0x80A10003), SystemServiceExports.SystemServiceReceiveEvent(context)); + } + [Fact] public void GetNoticeScreenSkipFlagWritesOneByteAtMemoryBoundary() { diff --git a/tests/SharpEmu.Libs.Tests/VideoOut/AsyncComputeBindingLayoutTests.cs b/tests/SharpEmu.Libs.Tests/VideoOut/AsyncComputeBindingLayoutTests.cs new file mode 100644 index 0000000..2320d95 --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/VideoOut/AsyncComputeBindingLayoutTests.cs @@ -0,0 +1,52 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Collections; +using System.Diagnostics; +using System.Runtime.CompilerServices; +using SharpEmu.Libs.Gpu.Pipelines; +using SharpEmu.Libs.Gpu.Rendering; +using SharpEmu.ShaderCompiler.Resources; +using Silk.NET.Vulkan; +using Xunit; + +namespace SharpEmu.Libs.Tests.VideoOut; + +public sealed class AsyncComputeBindingLayoutTests +{ + [Theory] + [InlineData(false)] + [InlineData(true)] + public void ACompletedAsyncPipelineRetainsItsGlobalDescriptorSet(bool usesBindlessImages) + { + // Exercise completion bookkeeping without calling Vulkan. Zero native + // handles keep the test independent of the driver's indexing features. + var type = PresenterUnderTest.PresenterType; + var presenter = RuntimeHelpers.GetUninitializedObject(type); + var pendingField = type.GetField("_pendingComputePipelines", PresenterUnderTest.InstanceMembers)!; + var pending = Activator.CreateInstance(pendingField.FieldType, nonPublic: true)!; + pendingField.SetValue(presenter, pending); + var pendingType = pendingField.FieldType.GenericTypeArguments[1]; + var completed = Activator.CreateInstance(pendingType, nonPublic: true)!; + pendingType.GetField("Compile")!.SetValue(completed, Task.FromResult(default(Pipeline))); + pendingType.GetField("StartTimestamp")!.SetValue(completed, Stopwatch.GetTimestamp()); + ((IDictionary)pending).Add(1ul, completed); + var entriesField = type.GetField("_pipelineEntries", PresenterUnderTest.InstanceMembers)!; + var entries = Activator.CreateInstance(entriesField.FieldType, nonPublic: true)!; + entriesField.SetValue(presenter, entries); + var description = new ComputePipelineDescription + { + Input = new ComputeInputInfo(), + Program = new ShaderProgram(1), + Stage = new ShaderProgramInfo + { + Bindings = BindingLayout.Allocate(new ShaderResourceInfo(), [], false, false, false, + usesBindlessImages: usesBindlessImages), + }, + }; + Assert.True(((IShaderPipelineHost)presenter).TryCreateComputePipeline(description, out var handle)); + var entry = ((IDictionary)entries)[handle.Pipeline]!; + Assert.Equal(usesBindlessImages, (bool)entry.GetType().GetField("UsesBindlessImages")!.GetValue(entry)!); + Assert.Empty((IDictionary)pending); + } +} diff --git a/tests/SharpEmu.Libs.Tests/VideoOut/HostVideoOptionsTests.cs b/tests/SharpEmu.Libs.Tests/VideoOut/HostVideoOptionsTests.cs index 4d81175..fd583fd 100644 --- a/tests/SharpEmu.Libs.Tests/VideoOut/HostVideoOptionsTests.cs +++ b/tests/SharpEmu.Libs.Tests/VideoOut/HostVideoOptionsTests.cs @@ -18,6 +18,7 @@ public sealed class HostVideoOptionsTests DisplayIndex = -2, RefreshRate = 5000, HdrMode = (HostHdrMode)99, + GuestResolution = (GuestDisplayResolution)99, }.Normalize(); Assert.Equal(640, options.Width); @@ -25,5 +26,42 @@ public sealed class HostVideoOptionsTests Assert.Equal(0, options.DisplayIndex); Assert.Equal(1000, options.RefreshRate); Assert.Equal(HostHdrMode.Auto, options.HdrMode); + Assert.Equal(GuestDisplayResolution.Hd, options.GuestResolution); + } + + [Theory] + [InlineData(0.5f, 0.5f)] + [InlineData(0.75f, 0.75f)] + [InlineData(1.0f, 1.0f)] + [InlineData(2.0f, 2.0f)] + [InlineData(0.0f, HostVideoOptions.MinimumRenderScale)] + [InlineData(-3.0f, HostVideoOptions.MinimumRenderScale)] + [InlineData(9.0f, HostVideoOptions.MaximumRenderScale)] + [InlineData(float.NaN, 1.0f)] + [InlineData(float.PositiveInfinity, 1.0f)] + [InlineData(0.333f, 0.33f)] + public void NormalizeClampsRenderScale(float requested, float expected) + { + Assert.Equal(expected, new HostVideoOptions { RenderScale = requested }.Normalize().RenderScale); + Assert.Equal(expected, HostVideoOptions.NormalizeRenderScale(requested)); + } + + [Fact] + public void RenderScaleDefaultsToNative() + { + Assert.Equal(1.0f, HostVideoOptions.Default.RenderScale); + Assert.Equal(1.0f, HostVideoOptions.Default.Normalize().RenderScale); + } + [Theory] + [InlineData(1920, 1080, GuestDisplayResolution.UltraHd)] + [InlineData(3840, 2160, GuestDisplayResolution.Hd)] + public void GuestModeIsIndependentOfWindowSize(int width, int height, GuestDisplayResolution mode) + { + var options = new HostVideoOptions { Width = width, Height = height, GuestResolution = mode }.Normalize(); + Assert.Equal(mode, options.GuestResolution); + Assert.Equal(GuestDisplayResolution.Hd, HostVideoOptions.Default.GuestResolution); + Span status = stackalloc byte[0x30]; + VideoOutExports.WriteOutputStatus(status, options.GuestResolution, 60, false); + Assert.Equal((int)mode, System.Buffers.Binary.BinaryPrimitives.ReadInt32LittleEndian(status)); } } diff --git a/tests/SharpEmu.Libs.Tests/VideoOut/PresenterImageBindingTests.cs b/tests/SharpEmu.Libs.Tests/VideoOut/PresenterImageBindingTests.cs index 5592b49..f7eed1c 100644 --- a/tests/SharpEmu.Libs.Tests/VideoOut/PresenterImageBindingTests.cs +++ b/tests/SharpEmu.Libs.Tests/VideoOut/PresenterImageBindingTests.cs @@ -45,6 +45,223 @@ public sealed class PresenterImageBindingTests : IClassFixture _vulkan = fixture.Vulkan; + [Fact] + public void PreparationPressure_KeepsStreamDataAndUsesFeedbackFromTheFinalTick() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var presenter = new PresenterUnderTest(_vulkan); + var harness = presenter.Harness; + var address = harness.MapBacked(0x10000, ReadWrite); + harness.Images.SetCollectionThresholds(ulong.MaxValue, ulong.MaxValue, ulong.MaxValue, 1); + var request = Color32(address); + var old = harness.Acquire(ref request); + var pool = (OptimalImageMemoryPool)GetFieldValue(harness.Images, "_imageMemoryPool"); + presenter.Run(() => + { + using var preparation = presenter.RenderHost.BeginPreparation(); + var stream = harness.Cache.GetUtilityBuffer(GpuBufferUsage.Stream); + Assert.True(stream.TryMap(16, out var offset)); + stream.Mapped.Slice((int)offset, 16).Fill(0xA5); + stream.Commit(); + var program = new ShaderProgramInfo { Stage = ShaderStageKind.Pixel, Hash = 99 }; + var stage = new ShaderStageResources(program, new ResourceSnapshot()); + var preparedType = PresenterType.GetNestedType("PreparedStageBindings", BindingFlags.NonPublic)!; + var prepared = Activator.CreateInstance(preparedType, InstanceMembers, null, [stage, program], null)!; + var before = harness.Scheduler.CurrentTick; + harness.Images.SetCollectionThresholds(0, 0, pool.AllocatedBytes - 1, 2); + var nextRequest = Color32(address + 256); + _ = harness.Images.FindImage(ref nextRequest); + Assert.True(harness.Scheduler.CurrentTick > before); + Assert.False(harness.Images.Contains(old)); + Assert.All(stream.Mapped.Slice((int)offset, 16).ToArray(), value => Assert.Equal((byte)0xA5, value)); + presenter.InvokeMethod("CommitRuntimeDescriptorBuffers", prepared); + var descriptors = presenter.InvokeMethod("RuntimeDescriptorsFor", program)!; + Assert.Equal(harness.Scheduler.CurrentTick, (ulong)GetFieldValue(descriptors, "MissScanTick")); + harness.Scheduler.Finish(); + presenter.InvokeMethod("DestroyRuntimeDescriptors"); + }); + } + + [Fact] + public void DrawBoundary_ReclaimsImagePressureBeforeAnotherFrameIsPresented() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var presenter = new PresenterUnderTest(_vulkan); + var harness = presenter.Harness; + var address = harness.MapBacked(0x10000, ReadWrite); + harness.Images.SetCollectionThresholds(ulong.MaxValue, ulong.MaxValue, ulong.MaxValue, 1); + var request = Color32(address); + var image = harness.Acquire(ref request); + harness.Images.SetCollectionThresholds(0, 0, 1, 2); + presenter.Run(() => presenter.RenderHost.RunPendingOperations()); + Assert.False(harness.Images.Contains(image)); + Assert.Equal(0UL, harness.Images.TotalUsedMemory); + var pool = (OptimalImageMemoryPool)GetFieldValue(harness.Images, "_imageMemoryPool"); + Assert.Equal(0UL, pool.AllocatedBytes); + } + + [Theory] + [InlineData(ShaderStageKind.Pixel, 2UL)] + [InlineData(ShaderStageKind.Compute, 1UL)] + public void RuntimeFeedback_DoesNotAcquireAnotherShadersTextures(ShaderStageKind nextStage, ulong nextHash) + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var presenter = new PresenterUnderTest(_vulkan); + var address = presenter.Harness.MapBacked(0x10000, ReadWrite); + var words = RegisterWords.Texture(address, GuestPixelFormat.Bits8_8_8_8UNorm, 4, 4); + var first = new ShaderProgramInfo { Stage = ShaderStageKind.Pixel, Hash = 1 }; + var next = new ShaderProgramInfo { Stage = nextStage, Hash = nextHash }; + presenter.Harness.Images.SetCollectionThresholds(ulong.MaxValue, ulong.MaxValue, ulong.MaxValue, 1); + presenter.Run(() => + { + var descriptors = presenter.InvokeMethod("RuntimeDescriptorsFor", first)!; + var misses = (System.Collections.Concurrent.ConcurrentQueue<(bool Image, uint[] Key)>)GetFieldValue(descriptors, "Misses"); + misses.Enqueue((true, [(uint)ImageDimension.Dim2D, .. words])); + var preparedType = PresenterType.GetNestedType("PreparedStageBindings", BindingFlags.NonPublic)!; + object Prepare(ShaderProgramInfo program) + { + var stage = new ShaderStageResources(program, new ResourceSnapshot()); + var prepared = Activator.CreateInstance(preparedType, InstanceMembers, null, [stage, program], null)!; + presenter.InvokeMethod("PrepareRuntimeDescriptors", prepared); + return prepared; + } + var firstBindings = Prepare(first); + presenter.Harness.Images.SetCollectionThresholds(ulong.MaxValue, ulong.MaxValue, ulong.MaxValue, 2); + var nextBindings = Prepare(next); + var images = (IList)firstBindings.GetType().GetProperty("RuntimeImages")!.GetValue(firstBindings)!; + var unrelatedImages = (IList)nextBindings.GetType().GetProperty("RuntimeImages")!.GetValue(nextBindings)!; + Assert.Single(images.Cast()); + Assert.Empty(unrelatedImages.Cast()); + var oldImage = (ResourceSlotIdentifier)GetFieldValue(images[0]!, "ImageIdentifier"); + presenter.Harness.Images.SetCollectionThresholds(0, 0, 1, 2); + presenter.RenderHost.RunPendingOperations(); + Assert.False(presenter.Harness.Images.Contains(oldImage)); + Assert.Equal(0UL, presenter.Harness.Images.TotalUsedMemory); + // Returning to the first program must retain its descriptor knowledge. + presenter.Harness.Images.SetCollectionThresholds(ulong.MaxValue, ulong.MaxValue, ulong.MaxValue, 3); + var resumed = Prepare(first); + Assert.Single(((IList)resumed.GetType().GetProperty("RuntimeImages")!.GetValue(resumed)!).Cast()); + presenter.InvokeMethod("DestroyRuntimeDescriptors"); + }); + } + + // A runtime descriptor read from registers may name memory that is not mapped (a lane whose + // result is unused, or a range the title has since freed). It stays out of the table, so the + // shader keeps reading the null descriptor, instead of creating an image over no memory. + [Fact] + public void RuntimeImageMissOverUnbackedMemory_StaysOutOfTheTable() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var presenter = new PresenterUnderTest(_vulkan); + var (_, hole, _) = presenter.Harness.MapBackedSandwich(); + var words = RegisterWords.Texture(hole, GuestPixelFormat.Bits32UInt, 1, 1); + var program = new ShaderProgramInfo { Stage = ShaderStageKind.Compute, Hash = 0x53 }; + presenter.Run(() => + { + var descriptors = presenter.InvokeMethod("RuntimeDescriptorsFor", program)!; + var misses = (System.Collections.Concurrent.ConcurrentQueue<(bool Image, uint[] Key)>)GetFieldValue(descriptors, "Misses"); + misses.Enqueue((true, [RuntimeDescriptorTable.ViewClass(ImageDimension.Dim2D, ImageNumericClass.Uint), .. words])); + var preparedType = PresenterType.GetNestedType("PreparedStageBindings", BindingFlags.NonPublic)!; + var stage = new ShaderStageResources(program, new ResourceSnapshot()); + var prepared = Activator.CreateInstance(preparedType, InstanceMembers, null, [stage, program], null)!; + + presenter.InvokeMethod("PrepareRuntimeDescriptors", prepared); + + Assert.Empty((IList)prepared.GetType().GetProperty("RuntimeImages")!.GetValue(prepared)!); + presenter.InvokeMethod("DestroyRuntimeDescriptors"); + }); + } + + // A runtime descriptor that describes no valid view reads as the null texture on the + // hardware: it stays out of the table instead of stopping the emulator. + [Fact] + public void RuntimeImageMissWithoutAValidView_StaysOutOfTheTable() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var fatal = new FatalScope(); + using var presenter = new PresenterUnderTest(_vulkan); + var address = presenter.Harness.MapBacked(0x10000, ReadWrite); + var words = RegisterWords.Texture(address, GuestPixelFormat.Bits32UInt, 1, 1); + words[3] &= 0x0FFF_FFFFu; + var program = new ShaderProgramInfo { Stage = ShaderStageKind.Compute, Hash = 0x55 }; + presenter.Run(() => + { + var descriptors = presenter.InvokeMethod("RuntimeDescriptorsFor", program)!; + var misses = (System.Collections.Concurrent.ConcurrentQueue<(bool Image, uint[] Key)>)GetFieldValue(descriptors, "Misses"); + misses.Enqueue((true, [RuntimeDescriptorTable.ViewClass(ImageDimension.Dim2D, ImageNumericClass.Uint), .. words])); + var preparedType = PresenterType.GetNestedType("PreparedStageBindings", BindingFlags.NonPublic)!; + var stage = new ShaderStageResources(program, new ResourceSnapshot()); + var prepared = Activator.CreateInstance(preparedType, InstanceMembers, null, [stage, program], null)!; + + presenter.InvokeMethod("PrepareRuntimeDescriptors", prepared); + + Assert.Empty((IList)prepared.GetType().GetProperty("RuntimeImages")!.GetValue(prepared)!); + presenter.InvokeMethod("DestroyRuntimeDescriptors"); + }); + } + + // A registered runtime entry outlives the memory it names when the title frees it. The next + // draw drops the entry instead of creating its image over memory that no longer exists. + [Fact] + public void RuntimeImageEntryOverFreedMemory_LeavesTheTable() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var presenter = new PresenterUnderTest(_vulkan); + var address = presenter.Harness.MapBacked(0x10000, ReadWrite); + var words = RegisterWords.Texture(address, GuestPixelFormat.Bits32UInt, 1, 1); + var program = new ShaderProgramInfo { Stage = ShaderStageKind.Compute, Hash = 0x54 }; + presenter.Run(() => + { + var descriptors = presenter.InvokeMethod("RuntimeDescriptorsFor", program)!; + var misses = (System.Collections.Concurrent.ConcurrentQueue<(bool Image, uint[] Key)>)GetFieldValue(descriptors, "Misses"); + misses.Enqueue((true, [RuntimeDescriptorTable.ViewClass(ImageDimension.Dim2D, ImageNumericClass.Uint), .. words])); + var preparedType = PresenterType.GetNestedType("PreparedStageBindings", BindingFlags.NonPublic)!; + var stage = new ShaderStageResources(program, new ResourceSnapshot()); + var prepared = Activator.CreateInstance(preparedType, InstanceMembers, null, [stage, program], null)!; + presenter.InvokeMethod("PrepareRuntimeDescriptors", prepared); + Assert.Single((IList)prepared.GetType().GetProperty("RuntimeImages")!.GetValue(prepared)!); + + presenter.Harness.ImageStore.Unregister(address, 0x10000); + Assert.True(presenter.Harness.Memory.TryUnmapBacked(address, 0x10000)); + var next = Activator.CreateInstance(preparedType, InstanceMembers, null, [stage, program], null)!; + presenter.InvokeMethod("PrepareRuntimeDescriptors", next); + + Assert.Empty((IList)next.GetType().GetProperty("RuntimeImages")!.GetValue(next)!); + Assert.Empty((IDictionary)GetFieldValue(descriptors, "Images")); + presenter.InvokeMethod("DestroyRuntimeDescriptors"); + }); + } + + // A runtime descriptor whose base level lies past its mips reads as the null texture, which + // the shader fetches through its unsigned heap array: the placeholder must be unsigned too. + [Fact] + public void RuntimeImageLoadMissPastTheLastLevel_BindsAnUnsignedNullTexture() + { + if (!GatePrerequisites.Ready(_vulkan)) return; + using var presenter = new PresenterUnderTest(_vulkan); + var address = presenter.Harness.MapBacked(0x10000, ReadWrite); + var words = RegisterWords.Texture(address, GuestPixelFormat.Bits32UInt, 1, 1); + words[3] |= 1u << 12; + var program = new ShaderProgramInfo { Stage = ShaderStageKind.Compute, Hash = 0x52 }; + presenter.Run(() => + { + var descriptors = presenter.InvokeMethod("RuntimeDescriptorsFor", program)!; + var misses = (System.Collections.Concurrent.ConcurrentQueue<(bool Image, uint[] Key)>)GetFieldValue(descriptors, "Misses"); + misses.Enqueue((true, [RuntimeDescriptorTable.ViewClass(ImageDimension.Dim2D, ImageNumericClass.Uint), .. words])); + var preparedType = PresenterType.GetNestedType("PreparedStageBindings", BindingFlags.NonPublic)!; + var stage = new ShaderStageResources(program, new ResourceSnapshot()); + var prepared = Activator.CreateInstance(preparedType, InstanceMembers, null, [stage, program], null)!; + + presenter.InvokeMethod("PrepareRuntimeDescriptors", prepared); + + var images = (IList)prepared.GetType().GetProperty("RuntimeImages")!.GetValue(prepared)!; + var binding = Assert.Single(images.Cast()); + var request = (ImageRequest)GetFieldValue(binding, "Request"); + Assert.Equal(Format.R32Uint, request.View.Format); + presenter.InvokeMethod("DestroyRuntimeDescriptors"); + }); + } + [Theory] [InlineData(GuestPixelFormat.Bits16UNorm, Format.D16Unorm)] [InlineData(GuestPixelFormat.Bits32Float, Format.D32Sfloat)] @@ -199,7 +416,8 @@ public sealed class PresenterImageBindingTests : IClassFixture Assert.False(image.Backing.Exists)); + Assert.Same(workingImages[0], workingImages[1]); + Assert.All(workingImages, image => Assert.True(image.Backing.Exists)); var stencil = harness.ReadImageBytes(attachment, ImageAspectFlags.StencilBit); for (var layer = 0; layer < layers; layer++) for (var row = 0; row < 64; row++) @@ -210,7 +428,14 @@ public sealed class PresenterImageBindingTests : IClassFixture(); + harness.Images.BindlessImageInvalidator = views => invalidated.AddRange(views); + presenter.Run(() => ((SharpEmu.HLE.GpuMemory.IGuestImageStore)harness.Images).Unregister(attachment.Description.Data.Address, attachment.Description.Data.Size)); + Assert.NotEmpty(workingViews); + Assert.All(workingViews, view => Assert.Contains(view, invalidated)); harness.Shutdown(); + Assert.All(workingImages, image => Assert.False(image.Backing.Exists)); } private static byte[] CreateStencilIncrementShader(uint imageBinding) diff --git a/tests/SharpEmu.Libs.Tests/VideoOut/PresenterStartupTests.cs b/tests/SharpEmu.Libs.Tests/VideoOut/PresenterStartupTests.cs index 51452fa..5b861c5 100644 --- a/tests/SharpEmu.Libs.Tests/VideoOut/PresenterStartupTests.cs +++ b/tests/SharpEmu.Libs.Tests/VideoOut/PresenterStartupTests.cs @@ -14,6 +14,54 @@ namespace SharpEmu.Libs.Tests.VideoOut; [Collection(SchedulingStateCollection.Name)] public sealed class PresenterStartupTests { + [Theory] + [InlineData(1L, 2L, 0UL, true)] + [InlineData(1L, 2L, 1UL, true)] + [InlineData(2L, 1L, 1UL, false)] + public void EarlierVideoFrameIsNotDiscardedByALaterGuestFlip( + long videoSequence, long guestSequence, ulong requiredTick, bool canTake) + { + using var state = new StartupState(); + var owner = typeof(VulkanVideoPresenter); + var presentationType = owner.GetNestedType("Presentation", BindingFlags.NonPublic)!; + var presenterType = owner.GetNestedType("Presenter", BindingFlags.NonPublic)!; + var presenter = RuntimeHelpers.GetUninitializedObject(presenterType); + object Make(long sequence, ulong tick) => Activator.CreateInstance(presentationType, + new object?[] { new byte[4], 1u, 1u, sequence, SharpEmu.ShaderCompiler.GuestDrawKind.None, + false, 0UL, 0L, false, tick, 0UL })!; + var video = owner.GetField("_pendingVideoPresentations", BindingFlags.Static | BindingFlags.NonPublic)!.GetValue(null)!; + var guest = owner.GetField("_pendingGuestImagePresentations", BindingFlags.Static | BindingFlags.NonPublic)!.GetValue(null)!; + var savedVideo = (Array)video.GetType().GetMethod("ToArray")!.Invoke(video, null)!; + var savedGuest = (Array)guest.GetType().GetMethod("ToArray")!.Invoke(guest, null)!; + void Clear(object queue) => queue.GetType().GetMethod("Clear")!.Invoke(queue, null); + void Enqueue(object queue, object frame) => queue.GetType().GetMethod("Enqueue")!.Invoke(queue, new[] { frame }); + try + { + Clear(video); + Clear(guest); + Enqueue(video, Make(videoSequence, 0)); + Enqueue(guest, Make(guestSequence, requiredTick)); + var ready = (bool)presenterType.GetMethod("HasReadyPresentationLocked", BindingFlags.Instance | BindingFlags.NonPublic)!.Invoke(presenter, null)!; + Assert.Equal(canTake, ready); + var args = new object?[] { null }; + var took = (bool)presenterType.GetMethod("TryTakePresentation", BindingFlags.Instance | BindingFlags.NonPublic)!.Invoke(presenter, args)!; + Assert.Equal(canTake, took); + if (took) + { + Assert.Equal(videoSequence, presentationType.GetProperty("Sequence")!.GetValue(args[0])); + Assert.Equal(1, guest.GetType().GetProperty("Count")!.GetValue(guest)); + Assert.Equal(0, video.GetType().GetProperty("Count")!.GetValue(video)); + } + } + finally + { + Clear(video); + Clear(guest); + foreach (var frame in savedVideo) Enqueue(video, frame!); + foreach (var frame in savedGuest) Enqueue(guest, frame!); + } + } + private sealed class StartupState : IDisposable { private static readonly Type PresenterType = typeof(VulkanVideoPresenter); diff --git a/tests/SharpEmu.Libs.Tests/VideoOut/PresenterUnderTest.cs b/tests/SharpEmu.Libs.Tests/VideoOut/PresenterUnderTest.cs index 6548b68..be37807 100644 --- a/tests/SharpEmu.Libs.Tests/VideoOut/PresenterUnderTest.cs +++ b/tests/SharpEmu.Libs.Tests/VideoOut/PresenterUnderTest.cs @@ -77,7 +77,7 @@ internal sealed class PresenterUnderTest : IDisposable { "_batchResources", "_batchRetireBuffers", "_pendingGuestSubmissions", "_deferredGuestImageVersionDestroys", - "_pipelineEntries", "_shaderModules", "_shaderModuleSpirvBytes", + "_pipelineEntries", "_shaderModules", "_shaderModuleSpirvBytes", "_runtimeDescriptorSets", "_preparedTextures", "_barriersAfterRendering", "_feedbackSnapshotPool", }) { diff --git a/tests/SharpEmu.Libs.Tests/VideoOut/RenderHostDeviceTests.BufferTail.cs b/tests/SharpEmu.Libs.Tests/VideoOut/RenderHostDeviceTests.BufferTail.cs new file mode 100644 index 0000000..d00485b --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/VideoOut/RenderHostDeviceTests.BufferTail.cs @@ -0,0 +1,191 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.Libs.Gpu.Pipelines; +using SharpEmu.Libs.Gpu.Rendering; +using SharpEmu.Libs.Gpu.Vulkan; +using SharpEmu.ShaderCompiler; +using SharpEmu.ShaderCompiler.Resources; +using SharpEmu.ShaderCompiler.Vulkan; +using Silk.NET.Vulkan; +using Xunit; +using static SharpEmu.Libs.Tests.Gpu.Images.ImageCacheTestSupport; +using static SharpEmu.ShaderCompiler.Tests.Resources.ResourceTestProgram; + +namespace SharpEmu.Libs.Tests.VideoOut; + +public sealed unsafe partial class RenderHostDeviceTests +{ + [Theory] + [InlineData(41, 0, false)] + [InlineData(41, 1, false)] + [InlineData(41, 0, true)] + [InlineData(58, 0, false)] + [InlineData(102, 0, false)] + [InlineData(93, 0, false)] + [InlineData(94, 0, false)] + [InlineData(95, 0, false)] + [InlineData(95, 1, false)] + [InlineData(93, 0, true)] + [InlineData(94, 0, true)] + [InlineData(95, 0, true)] + [InlineData(95, 1, true)] + [InlineData(95, 0, false, false)] + [InlineData(95, 1, false, false)] + [InlineData(95, 0, true, false)] + [InlineData(95, 1, true, false)] + public void FormattedByteBuffer_ReadsItsTailWithoutExposingPadding(int bytes, int bias, bool zeroOutOfBounds, bool formatted = true) + { + if (!Ready() || !_vulkan!.ShaderInt64) return; + using var presenter = new PresenterUnderTest(_vulkan!); + presenter.LoadRenderingCommands(); + presenter.SetField("_minStorageBufferOffsetAlignment", 256UL); + var harness = presenter.Harness; + var input = harness.MapBacked(0x10000, ReadWrite) + (ulong)bias; + var output = harness.MapBacked(0x10000, ReadWrite); + var states = Enumerable.Repeat((byte)0x81, bytes + 4).ToArray(); + states.AsSpan(bytes - 3, 3).Fill(8); + states.AsSpan(bytes).Fill(0x42); // Outside the guest descriptor, inside its backing. + harness.Write(input, states); + + var firstIndex = Math.Max(0, bytes - 32); + var guest = Program( + Vop2(0, "VAddU32", 1, Operand((uint)firstIndex), Gen5Operand.Vector(0)), + BufferAccess(4, formatted ? "BufferLoadFormatX" : "BufferLoadSbyte", 0, vectorData: 4, indexEnabled: true, vectorAddress: 1), + BufferAccess(12, "BufferStoreByte", 0, vectorData: 4, indexEnabled: true, vectorAddress: 1), + Vop2(20, "VLshlrevB32", 2, Operand(2), Gen5Operand.Vector(0)), + BufferAccess(24, "BufferStoreDword", 4, vectorData: 4, offsetEnabled: true, vectorAddress: 2), + EndProgram(32)); + var plan = ShaderResourcePlan.Extract(guest, ShaderStage.Compute, 1, 0, 8); + var resources = ResourceMaterializer.ApplyTo(plan, ResourceSpecialization.Default(plan.Info)); + resources.Info.Buffers[0].DescriptorFormat = 6; // R8 SINT, as in Yotei's instance-state buffer. + resources.Info.Buffers[0].DescriptorSwizzle = DescriptorConstants.IdentityDestinationSelect; + var layout = BindingLayout.Allocate(resources.Info, BindingLayout.CollectUserDataRegisters(guest, 0, 8), false, false, false, usesRuntimeBufferStrides: true); + var request = new ShaderCompileRequest(plan, resources, layout) { LocalSizeX = 64, ZeroOutOfBoundsBufferReads = zeroOutOfBounds }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var compiled, out var error), error); + var program = new ShaderProgramInfo { Stage = ShaderStageKind.Compute, Hash = 0xB001, UserDataCount = 8, Resources = resources, Bindings = layout }; + var snapshot = new ResourceSnapshot + { + UserData = new uint[8], + Buffers = + [ + [(uint)input, (uint)(input >> 32) | (1u << 16), (uint)bytes, (6u << 12) | DescriptorConstants.IdentityDestinationSelect], + [(uint)output, (uint)(output >> 32), 256, (20u << 12) | DescriptorConstants.IdentityDestinationSelect], + ], + }; + presenter.Run(() => + { + var host = (IShaderPipelineHost)presenter.Instance; + var shader = new ShaderProgram(program.Hash, host.CreateShaderModule(new VulkanCompiledGuestShader(compiled.Spirv), ShaderStage.Compute, program.Hash, program.Hash)); + var stage = new ShaderStageResources(program, snapshot); + var pipeline = host.CreateComputePipeline(new ComputePipelineDescription + { + Input = new ComputeInputInfo { ThreadsX = 64, ThreadsY = 1, ThreadsZ = 1, Stage = stage }, + Program = shader, Stage = program, + }); + using var preparation = presenter.RenderHost.BeginPreparation(); + // Keep the owner so bias=1 exercises a genuinely unaligned native view, + // rather than the aligned temporary upload used for a first CPU write. + _ = harness.Cache.ObtainBuffer(input, (ulong)bytes, false, requiresDeviceAddress: true); + var prepared = presenter.RenderHost.PrepareBindings(stage); + presenter.RenderHost.BindResources(prepared); + presenter.RenderHost.CommitBindings(PipelineBindPoint.Compute, in pipeline, [prepared]); + presenter.RenderHost.BindPipeline(PipelineBindPoint.Compute, in pipeline); + presenter.RenderHost.Dispatch(1, 1, 1); + }); + presenter.Run(() => presenter.InvokeMethod("FlushBatchedGuestCommands")); + harness.Finish(); + var actual = new byte[256]; + presenter.Run(() => + { + var host = (IShaderPipelineHost)presenter.Instance; + for (var word = 0; word < 64; word++) + { + Assert.True(host.TryReadGuestWord(output + (ulong)word * 4, out var value)); + BitConverter.TryWriteBytes(actual.AsSpan(word * 4), value); + } + }); + for (var lane = 0; lane < 64; lane++) + { + var index = lane + firstIndex; + var expected = index < bytes ? unchecked((uint)(int)(sbyte)states[index]) : 0u; + Assert.True(expected == BitConverter.ToUInt32(actual, lane * 4), $"bytes={bytes} bias={bias} index={index}: expected {expected:X8}, got {BitConverter.ToUInt32(actual, lane * 4):X8}"); + } + var owner = presenter.Run(() => harness.Cache.GetBuffer(harness.Cache.FindBuffer(input, (ulong)states.Length))); + Assert.Equal(states, harness.ReadBack(owner, owner.Offset(input), (ulong)states.Length)); + harness.Shutdown(); + } + + // The hardware range-checks a buffer access by its first byte, so a dword load that starts + // inside a guest range ending mid-word returns the whole word; the next word reads zero. + [Theory] + [InlineData(93, false)] + [InlineData(94, false)] + [InlineData(95, false)] + [InlineData(96, false)] + [InlineData(95, true)] + public void DwordBuffer_ReadsAWordThatStartsInsideTheRange(int bytes, bool zeroOutOfBounds) + { + if (!Ready() || !_vulkan!.ShaderInt64) return; + using var presenter = new PresenterUnderTest(_vulkan!); + presenter.LoadRenderingCommands(); + presenter.SetField("_minStorageBufferOffsetAlignment", 256UL); + var harness = presenter.Harness; + var input = harness.MapBacked(0x10000, ReadWrite); + var output = harness.MapBacked(0x10000, ReadWrite); + var states = Enumerable.Range(1, bytes + 4).Select(value => (byte)value).ToArray(); + harness.Write(input, states); + + var guest = Program( + Vop2(0, "VLshlrevB32", 2, Operand(2), Gen5Operand.Vector(0)), + BufferAccess(4, "BufferLoadDword", 0, vectorData: 4, offsetEnabled: true, vectorAddress: 2), + BufferAccess(12, "BufferStoreDword", 4, vectorData: 4, offsetEnabled: true, vectorAddress: 2), + EndProgram(20)); + var plan = ShaderResourcePlan.Extract(guest, ShaderStage.Compute, 1, 0, 8); + var resources = ResourceMaterializer.ApplyTo(plan, ResourceSpecialization.Default(plan.Info)); + var layout = BindingLayout.Allocate(resources.Info, BindingLayout.CollectUserDataRegisters(guest, 0, 8), false, false, false, usesRuntimeBufferStrides: true); + var request = new ShaderCompileRequest(plan, resources, layout) { LocalSizeX = 64, ZeroOutOfBoundsBufferReads = zeroOutOfBounds }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var compiled, out var error), error); + var program = new ShaderProgramInfo { Stage = ShaderStageKind.Compute, Hash = 0xB002, UserDataCount = 8, Resources = resources, Bindings = layout }; + var snapshot = new ResourceSnapshot + { + UserData = new uint[8], + Buffers = + [ + [(uint)input, (uint)(input >> 32), (uint)bytes, (20u << 12) | DescriptorConstants.IdentityDestinationSelect], + [(uint)output, (uint)(output >> 32), 256, (20u << 12) | DescriptorConstants.IdentityDestinationSelect], + ], + }; + presenter.Run(() => + { + var host = (IShaderPipelineHost)presenter.Instance; + var shader = new ShaderProgram(program.Hash, host.CreateShaderModule(new VulkanCompiledGuestShader(compiled.Spirv), ShaderStage.Compute, program.Hash, program.Hash)); + var stage = new ShaderStageResources(program, snapshot); + var pipeline = host.CreateComputePipeline(new ComputePipelineDescription + { + Input = new ComputeInputInfo { ThreadsX = 64, ThreadsY = 1, ThreadsZ = 1, Stage = stage }, + Program = shader, Stage = program, + }); + using var preparation = presenter.RenderHost.BeginPreparation(); + _ = harness.Cache.ObtainBuffer(input, (ulong)bytes, false, requiresDeviceAddress: true); + var prepared = presenter.RenderHost.PrepareBindings(stage); + presenter.RenderHost.BindResources(prepared); + presenter.RenderHost.CommitBindings(PipelineBindPoint.Compute, in pipeline, [prepared]); + presenter.RenderHost.BindPipeline(PipelineBindPoint.Compute, in pipeline); + presenter.RenderHost.Dispatch(1, 1, 1); + }); + presenter.Run(() => presenter.InvokeMethod("FlushBatchedGuestCommands")); + harness.Finish(); + presenter.Run(() => + { + var host = (IShaderPipelineHost)presenter.Instance; + for (var lane = 0; lane < 64; lane++) + { + Assert.True(host.TryReadGuestWord(output + (ulong)lane * 4, out var value)); + var expected = lane * 4 < bytes ? BitConverter.ToUInt32(states, lane * 4) : 0u; + Assert.True(expected == value, $"bytes={bytes} lane={lane}: expected {expected:X8}, got {value:X8}"); + } + }); + harness.Shutdown(); + } +} diff --git a/tests/SharpEmu.Libs.Tests/VideoOut/RenderHostDeviceTests.CompressedExports.cs b/tests/SharpEmu.Libs.Tests/VideoOut/RenderHostDeviceTests.CompressedExports.cs new file mode 100644 index 0000000..976d55f --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/VideoOut/RenderHostDeviceTests.CompressedExports.cs @@ -0,0 +1,60 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.Libs.Gpu.Pipelines; +using SharpEmu.Libs.Gpu.Rendering; +using SharpEmu.Libs.Tests.Gpu.Images; +using SharpEmu.ShaderCompiler; +using SharpEmu.ShaderCompiler.Resources; +using SharpEmu.ShaderCompiler.Vulkan; +using Xunit; +using static SharpEmu.Libs.Tests.Gpu.Images.ImageCacheTestSupport; +using static SharpEmu.ShaderCompiler.Tests.Resources.ResourceTestProgram; + +namespace SharpEmu.Libs.Tests.VideoOut; + +public sealed unsafe partial class RenderHostDeviceTests +{ + [Theory] + [InlineData(Gen5PixelExportFormat.Float16, 0x3A003400u, 0x3C003800u, 64, 191, 128)] + [InlineData(Gen5PixelExportFormat.Unorm16, 0xBFFF4000u, 0xFFFF8000u, 64, 191, 128)] + [InlineData(Gen5PixelExportFormat.Snorm16, 0x5FFF2000u, 0x7FFF4000u, 64, 191, 128)] + [InlineData(Gen5PixelExportFormat.Uint16, 0x00000001u, 0x00010001u, 255, 0, 255)] + [InlineData(Gen5PixelExportFormat.Sint16, 0xFFFF0001u, 0x00010001u, 255, 0, 255)] + public void CompressedExport_DrawDecodesTheShaderColorFormat( + Gen5PixelExportFormat format, uint low, uint high, int red, int green, int blue) + { + if (!Ready()) return; + using var presenter = new PresenterUnderTest(_vulkan); + presenter.LoadRenderingCommands(); + var harness = presenter.Harness; + var target = harness.MapBacked(0x10000, ReadWrite); + var vertices = harness.MapBacked(0x10000, ReadWrite); + var words = RegisterWords.Color(target, Size, Size); + harness.Write(vertices, Triangle(-1f, -1f, 3f, -1f, -1f, 3f)); + var export = new Gen5ShaderInstruction(8, Gen5ShaderEncoding.Exp, "Exp", [], + [Gen5Operand.Vector(0), Gen5Operand.Vector(1), Gen5Operand.Vector(0), Gen5Operand.Vector(0)], + [], new Gen5ExportControl(0, 15, true, true, true)); + var (plan, resources, layout) = Prepare(Program( + MoveVector(0, 0, low), MoveVector(4, 1, high), export, EndProgram(16)), ShaderStage.Pixel, userDataCount: 0); + var request = new ShaderCompileRequest(plan, resources, layout) + { + PixelOutputs = [new Gen5PixelOutputBinding(0, 0, Gen5PixelOutputKind.Float) { ExportFormat = format }], + EnableGraphicsSubgroupOperations = false, + }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + var provider = new FixedProgramProvider((IShaderPipelineHost)presenter.Instance, vertices, + interpolationShader: shader.Spirv); + var executor = new RenderExecutor(presenter.RenderHost, provider); + presenter.Run(() => executor.DrawAuto(1, Banks(words), Draw())); + presenter.Run(() => presenter.InvokeMethod("FlushBatchedGuestCommands")); + harness.Finish(); + var actual = Pixel(harness.ReadImageBytes(TargetImage(presenter, words)), Size / 2, Size / 2); + Assert.InRange((int)(actual & 255), red - 1, red + 1); + Assert.InRange((int)((actual >> 8) & 255), green - 1, green + 1); + Assert.InRange((int)((actual >> 16) & 255), blue - 1, blue + 1); + Assert.Equal(255u, actual >> 24); + harness.Shutdown(); + _vulkan.AssertNoValidationMessages(); + } +} diff --git a/tests/SharpEmu.Libs.Tests/VideoOut/RenderHostDeviceTests.SubgroupLanes.cs b/tests/SharpEmu.Libs.Tests/VideoOut/RenderHostDeviceTests.SubgroupLanes.cs new file mode 100644 index 0000000..6a86856 --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/VideoOut/RenderHostDeviceTests.SubgroupLanes.cs @@ -0,0 +1,115 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.Libs.Gpu.Pipelines; +using SharpEmu.Libs.Gpu.Rendering; +using SharpEmu.Libs.Tests.Gpu.Images; +using SharpEmu.ShaderCompiler; +using SharpEmu.ShaderCompiler.Resources; +using SharpEmu.ShaderCompiler.Vulkan; +using Xunit; +using static SharpEmu.Libs.Tests.Gpu.Images.ImageCacheTestSupport; +using static SharpEmu.ShaderCompiler.Tests.Resources.ResourceTestProgram; + +namespace SharpEmu.Libs.Tests.VideoOut; + +public sealed unsafe partial class RenderHostDeviceTests +{ + // This is the wave-minimum idiom at 0x80003C5100/0x1CC, isolated from the + // texture loop. A tiny primitive leaves lane 31 absent from its fragment + // subgroup. Red exports every live lane's input, green exports the reduced + // result, so the CPU can compare it with the true minimum of the live lanes. + [Fact] + public void PixelWaveMinimum_UsesItsOwnValueForAbsentSubgroupLanes() + { + if (!Ready()) return; + using var presenter = new PresenterUnderTest(_vulkan); + presenter.LoadRenderingCommands(); + var harness = presenter.Harness; + var target = harness.MapBacked(0x10000, ReadWrite); + var vertices = harness.MapBacked(0x10000, ReadWrite); + var words = RegisterWords.Color(target, Size, Size); + // Four pixels at most: deliberately much smaller than an NVIDIA wave32. + harness.Write(vertices, Triangle(-1f, -1f, -0.875f, -1f, -1f, -0.875f)); + var provider = new FixedProgramProvider((IShaderPipelineHost)presenter.Instance, vertices, + interpolationShader: CompilePixelWaveMinimum()); + var executor = new RenderExecutor(presenter.RenderHost, provider); + presenter.Run(() => executor.DrawAuto(1, Banks(words), Draw())); + presenter.Run(() => presenter.InvokeMethod("FlushBatchedGuestCommands")); + harness.Finish(); + + var pixels = harness.ReadImageBytes(TargetImage(presenter, words)); + var covered = Enumerable.Range(0, checked((int)(Size * Size))) + .Select(index => BitConverter.ToUInt32(pixels, index * sizeof(uint))) + .Where(value => value >> 24 == byte.MaxValue) + .ToArray(); + Assert.InRange(covered.Length, 1, 31); + var trueMinimum = covered.Min(value => value & byte.MaxValue); + Assert.All(covered, value => + { + Assert.Equal(trueMinimum, (value >> 8) & byte.MaxValue); + Assert.Equal(byte.MaxValue, value >> 24); + }); + + harness.Shutdown(); + _vulkan.AssertNoValidationMessages(); + } + + private static byte[] CompilePixelWaveMinimum() + { + const uint One = 0x3F80_0000; + const uint OneOver32 = 0x3D00_0000; + var instructions = new List(); + void Add(Gen5ShaderInstruction instruction) => instructions.Add(instruction with { Pc = (uint)instructions.Count * 8 }); + + // v14 = subgroup invocation id + 1. It makes every present lane distinct. + Add(Vop2(0, "VMbcntLoU32B32", 14, Operand(uint.MaxValue), Operand(1))); + Add(Vopc(0, "VCmpEqU32", Gen5Operand.Vector(14), 14)); + Add(Sop2(0, "SAndB64", 6, Gen5Operand.Scalar(126), Gen5Operand.Scalar(106))); + // 0xBEEA287E: save EXEC and force it, exactly like the game's reduction. + Add(Sop1(0, "SOrn2SaveexecB64", 106, Gen5Operand.Scalar(126))); + Add(new Gen5ShaderInstruction(0, Gen5ShaderEncoding.Vop3, "VCndmaskB32", [0, 0], + [Operand(uint.MaxValue), Gen5Operand.Vector(14), Gen5Operand.Scalar(6)], [Gen5Operand.Vector(13)], + new Gen5Vop3Control(0, 0, 0, false, 0, null))); + foreach (var shift in new uint[] { 1, 2, 4, 8 }) + { + // 0x261A1AFA_FF011?0D: v_min_u32 v13, v13, v13 row_shr:shift. + Add(Vop2(0, "VMinU32", 13, Gen5Operand.Vector(13), Gen5Operand.Vector(13)) with + { + Control = new Gen5DppControl(0x110 + shift, false, false, 0, 0, 15, 15), + }); + } + + // 0xD778100C_0305830D: exchange the two rows, then reduce them. + Add(Vop3(0, "VPermlanex16B32", 12, Gen5Operand.Vector(13), Operand(uint.MaxValue), Operand(uint.MaxValue)) with + { + Control = new Gen5Vop3Control(0, 0, 0, false, 2, null), + }); + Add(Vop2(0, "VMinU32", 13, Gen5Operand.Vector(13), Gen5Operand.Vector(12))); + Add(Sop1(0, "SMovB64", 126, Gen5Operand.Scalar(106))); + Add(ReadLane(0, 6, 13, 31)); + Add(ReadLane(0, 7, 13, 63)); + Add(Sop2(0, "SMinU32", 8, Gen5Operand.Scalar(6), Gen5Operand.Scalar(7))); + // Red is the input, green is the shared reduction, both normalized by 32. + Add(Vop1(0, "VCvtF32U32", 0, Gen5Operand.Vector(14))); + Add(MoveVectorFromScalar(0, 1, 8)); + Add(Vop1(0, "VCvtF32U32", 1, Gen5Operand.Vector(1))); + Add(Vop2(0, "VMulF32", 0, Gen5Operand.Vector(0), Operand(OneOver32))); + Add(Vop2(0, "VMulF32", 1, Gen5Operand.Vector(1), Operand(OneOver32))); + Add(MoveVector(0, 2, One)); + Add(MoveVector(0, 3, One)); + Add(new Gen5ShaderInstruction(0, Gen5ShaderEncoding.Exp, "Exp", [], + [Gen5Operand.Vector(0), Gen5Operand.Vector(1), Gen5Operand.Vector(2), Gen5Operand.Vector(3)], + [], new Gen5ExportControl(0, 15, false, true, true))); + Add(EndProgram(0)); + + var (plan, resources, layout) = Prepare(Program([.. instructions]), ShaderStage.Pixel, userDataCount: 0); + var request = new ShaderCompileRequest(plan, resources, layout) + { + PixelOutputs = [new Gen5PixelOutputBinding(0, 0, Gen5PixelOutputKind.Float)], + EnableGraphicsSubgroupOperations = true, + }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + return shader.Spirv; + } +} diff --git a/tests/SharpEmu.Libs.Tests/VideoOut/RenderHostDeviceTests.Tessellation.cs b/tests/SharpEmu.Libs.Tests/VideoOut/RenderHostDeviceTests.Tessellation.cs new file mode 100644 index 0000000..1945630 --- /dev/null +++ b/tests/SharpEmu.Libs.Tests/VideoOut/RenderHostDeviceTests.Tessellation.cs @@ -0,0 +1,256 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.HLE.GpuMemory; +using SharpEmu.Libs.Gpu; +using SharpEmu.Libs.Gpu.Buffers; +using SharpEmu.Libs.Gpu.GpuCommands.Registers; +using SharpEmu.Libs.Gpu.GpuCommands; +using SharpEmu.Libs.Gpu.Vulkan; +using SharpEmu.Libs.Gpu.Images; +using SharpEmu.Libs.Gpu.Pipelines; +using SharpEmu.Libs.Gpu.Rendering; +using SharpEmu.Libs.Tests.Gpu.Images; +using SharpEmu.ShaderCompiler; +using SharpEmu.ShaderCompiler.Resources; +using SharpEmu.ShaderCompiler.Vulkan; +using Silk.NET.Vulkan; +using Xunit; +using static SharpEmu.ShaderCompiler.Tests.Resources.ResourceTestProgram; +using static SharpEmu.Libs.Tests.Gpu.Images.ImageCacheTestSupport; +using ResourceSnapshot = SharpEmu.ShaderCompiler.Resources.ResourceSnapshot; + +namespace SharpEmu.Libs.Tests.VideoOut; + +public sealed unsafe partial class RenderHostDeviceTests +{ + [Theory] + [InlineData(1f, true)] + [InlineData(0f, false)] + public void Tessellation_FactorsGenerateOrCullTheDomainPatch(float firstOuter, bool visible) => + RenderTessellationPatch(firstOuter, visible, false); + + [Theory] + [InlineData(1f, true)] + [InlineData(0f, false)] + public void Tessellation_MergedHullWritesFactorsConsumedByTheNativeDraw(float firstOuter, bool visible) => + RenderTessellationPatch(firstOuter, visible, true); + + [Fact] + public void Tessellation_SuccessiveBatchesReuseTheFactorRingAfterThePreviousDraw() => + RenderTessellationPatch(0, true, true, patchCount: 2); + + // With the guest's off-chip configuration, hull groups that fit run in one dispatch, each + // writing its factors to its own run of the ring, and one draw consumes them all. + [Fact] + public void Tessellation_GroupsInFlightWriteTheirOwnFactorRuns() + { + TessellationOffchip.Configure(0xFF); + try + { + RenderTessellationPatch(0, true, true, patchCount: 2, ringPatches: 2); + } + finally + { + TessellationOffchip.Reset(); + } + } + + [Fact] + public void TessellationOffchip_DecodesTheBufferCountAndGranularity() + { + Assert.Equal((256u, 32u * 1024), TessellationOffchip.Decode(0xFF)); + Assert.Equal((1u, 64u * 1024), TessellationOffchip.Decode(1u << 10)); + Assert.Equal((1024u, 256u * 1024), TessellationOffchip.Decode(0x3FFu | (3u << 10))); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void Tessellation_VertexOffsetUsesPacketRegistersOrIndirectArguments(bool indexed) => + RenderTessellationPatch(0, false, true, verifyOffset: true, indexed: indexed); + + private void RenderTessellationPatch(float firstOuter, bool visible, bool computeFactors, uint patchCount = 1, + bool verifyOffset = false, bool indexed = false, uint ringPatches = 1) + { + if (!Ready()) return; + using var presenter = new PresenterUnderTest(_vulkan!); + _vulkan!.Vk.GetPhysicalDeviceProperties(_vulkan.Physical, out var properties); + presenter.SetField("_minStorageBufferOffsetAlignment", properties.Limits.MinStorageBufferOffsetAlignment); + presenter.LoadRenderingCommands(); + var harness = presenter.Harness; + using var factors = new GpuBuffer(_vulkan!.DeviceInfo, harness.Scheduler, GpuBufferUsage.Upload, + 0, GpuBuffer.AllFlags | BufferUsageFlags.ShaderDeviceAddressBit, 24); + Span levels = [firstOuter, 1f, 1f, 1f, 1f, 1f]; + factors.Write(0, System.Runtime.InteropServices.MemoryMarshal.AsBytes(levels)); + var target = harness.MapBacked(0x10000, ReadWrite); + var guestFactors = computeFactors ? harness.MapBacked(0x10000, ReadWrite) : 0; + var words = RegisterWords.Color(target, Size, Size); + var executor = new RenderExecutor(presenter.RenderHost, + new TessellationProgramProvider((IShaderPipelineHost)presenter.Instance, factors.DeviceAddress, guestFactors, firstOuter, + patchCount > 1 || verifyOffset, verifyOffset ? 2u : 0u, 24 * ringPatches)); + GuestGpuMemoryHook.Attach(harness.Gpu); + try + { + var banks = Banks(words); + if (computeFactors) + { + banks.Context.ShaderStages = 1u | (1u << 2) | (1u << 3) | (1u << 8) | (1u << 13) | (1u << 25); + banks.Context.ShaderInterface.MinTessellationLevel = BitConverter.SingleToUInt32Bits(1); + banks.Context.ShaderInterface.MaxTessellationLevel = BitConverter.SingleToUInt32Bits(9); + // The guest registers match the provider: quads, equal spacing, counter-clockwise + // triangles, one patch of four input and four output control points per group. + banks.Context.ShaderInterface.TessellationFactorParameter = 2u | (0u << 2) | (3u << 5); + banks.Context.ShaderInterface.LocalHullConfiguration = 1u | (4u << 8) | (4u << 14); + } + if (indexed) + { + banks.UserConfig.IndexOffset = uint.MaxValue; + var indexAddress = harness.MapBacked(0x10000, ReadWrite); + harness.Write(indexAddress, Bytes((ushort)0, (ushort)1, (ushort)2, (ushort)3)); + presenter.Run(() => executor.DrawIndexed(1, banks, + new(0, 0, 4, indexAddress, (uint)GuestIndexType.Index16, 1, 1, 0, DrawOffsetSource.Packet))); + } + else + { + if (verifyOffset) banks.UserConfig.IndexOffset = 1; + presenter.Run(() => executor.DrawAuto(1, banks, + new(0, 0, 4 * patchCount, 1, 0, 0, + verifyOffset ? DrawOffsetSource.IndirectArguments : DrawOffsetSource.Packet))); + } + presenter.Run(() => presenter.InvokeMethod("FlushBatchedGuestCommands")); + harness.Finish(); + var image = harness.ReadImageBytes(TargetImage(presenter, words)); + Assert.Equal(visible ? Red : 0u, Pixel(image, Size / 2, Size / 2)); + Assert.Equal(visible ? Red : 0u, Pixel(image, 2, 2)); + if (ringPatches > 1) + { + // Patch 0 wrote its zero factor at the start of the ring; patch 1 wrote its + // own factors one patch further instead of over patch 0's. + var host = (IShaderPipelineHost)presenter.Instance; + uint Factor(ulong address) => presenter.Run(() => host.TryReadGuestWord(address, out var word) ? word : uint.MaxValue); + Assert.Equal(0u, Factor(guestFactors)); + Assert.Equal(BitConverter.SingleToUInt32Bits(1), Factor(guestFactors + 24)); + } + } + finally { GuestGpuMemoryHook.Attach(null); } + harness.Shutdown(); + } + + private sealed class TessellationProgramProvider(IShaderPipelineHost host, ulong factorAddress, ulong guestFactors, float firstOuter, + bool useAbsolutePatchId, uint factorSourceRegister, uint factorRingBytes) : IShaderPipelineProvider + { + private ShaderProgram _vertex, _control, _domain, _pixel; + private ShaderProgramInfo _domainInfo = null!; + private ComputeProgram? _hull; + private static readonly ShaderProgramInfo PixelInfo = FixedProgramProvider.EmptyProgram(ShaderStageKind.Pixel, 4); + + public GraphicsPrograms GetGraphicsPrograms(VertexStageRegisters vertex, PixelStageRegisters pixel, + ShaderInterfaceRegisters shaderInterface, ContextRegisters context, ReadOnlySpan mappings, + bool pixelActive, bool depthBound) + { + if (!_domain.IsValid) + { + var program = Program( + Vop2(0, "VMulF32", 10, Operand(BitConverter.SingleToUInt32Bits(2f)), Gen5Operand.Vector(5)), + Vop2(4, "VAddF32", 10, Operand(BitConverter.SingleToUInt32Bits(-1f)), Gen5Operand.Vector(10)), + Vop2(8, "VMulF32", 11, Operand(BitConverter.SingleToUInt32Bits(2f)), Gen5Operand.Vector(6)), + Vop2(12, "VAddF32", 11, Operand(BitConverter.SingleToUInt32Bits(-1f)), Gen5Operand.Vector(11)), + Vop1(16, "VMovB32", 12, Operand(0)), + Vop1(20, "VMovB32", 13, Operand(BitConverter.SingleToUInt32Bits(1f))), + new(24, Gen5ShaderEncoding.Exp, "Exp", [0u, 0u], + [Gen5Operand.Vector(10), Gen5Operand.Vector(11), Gen5Operand.Vector(12), Gen5Operand.Vector(13)], [], + new Gen5ExportControl(12, 15, false, true, false)), EndProgram(32)); + var plan = ShaderResourcePlan.Extract(program, ShaderStage.TessellationEvaluation, Hash, 8, 0, waveSize: 32); + var resources = ResourceMaterializer.ApplyTo(plan, ResourceSpecialization.Default(plan.Info)); + var layout = BindingLayout.Allocate(resources.Info, [], false, false, false, usesTessellationData: true); + var request = new ShaderCompileRequest(plan, resources, layout) + { Tessellation = new(Gen5TessellationDomain.Quads, Gen5TessellationSpacing.Equal, false, false) }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + _domainInfo = new() { Stage = ShaderStageKind.TessellationEvaluation, Hash = Hash, Resources = resources, Bindings = layout }; + _vertex = new(1, host.CreateShaderModule(new VulkanCompiledGuestShader(SpirvFixedShaders.CreateFullscreenVertex(0)), ShaderStage.Vertex, 1, 1)); + _control = new(2, host.CreateShaderModule(new VulkanCompiledGuestShader(Gen5TessellationBridge.CompileControl(layout, Gen5TessellationDomain.Quads).Spirv), ShaderStage.TessellationEvaluation, 2, 2)); + _domain = new(3, host.CreateShaderModule(new VulkanCompiledGuestShader(shader.Spirv), ShaderStage.TessellationEvaluation, Hash, 3)); + _pixel = new(4, host.CreateShaderModule(new VulkanCompiledGuestShader(SpirvFixedShaders.CreateSolidFragment(1, 0, 0, 1)), ShaderStage.Pixel, 4, 4)); + if (guestFactors != 0) _hull = CreateHull(); + } + var data = new uint[Gen5TessellationData.DwordCount]; + data[Gen5TessellationData.PatchCount] = 1; + data[Gen5TessellationData.FactorAddress] = (uint)factorAddress; + data[Gen5TessellationData.FactorAddress + 1] = (uint)(factorAddress >> 32); + data[Gen5TessellationData.FactorBytes] = 24; + data[Gen5TessellationData.MinimumLevel] = BitConverter.SingleToUInt32Bits(1); + data[Gen5TessellationData.MaximumLevel] = BitConverter.SingleToUInt32Bits(9); + return new() + { + Vertex = _vertex, Pixel = _pixel, + Tessellation = _hull is null ? null : new(new(Gen5TessellationDomain.Quads, Gen5TessellationSpacing.Equal, false, false), new(4, 4, 1, 0), _hull), + VertexInput = new() { Tessellation = new(_control, _domain, 4), Stage = new(_domainInfo, new ResourceSnapshot()) { TessellationData = data } }, + PixelInput = new() { Stage = new(PixelInfo, new ResourceSnapshot()) }, + }; + } + + private ComputeProgram CreateHull() + { + var instructions = new List + { + Vop2(0, "VLshlrevB32", 4, Operand(2), Gen5Operand.Vector(3)), + Vop1(4, "VMovB32", 10, Gen5Operand.Scalar(20)), + DataShare(8, "DsWriteB32", false, [Gen5Operand.Vector(4), Gen5Operand.Vector(10)], []), + new(16, Gen5ShaderEncoding.Sopp, "SBarrier", [0u], [], [], null), + DataShare(20, "DsReadB32", false, [Gen5Operand.Vector(4)], [10]), + Vopc(28, "VCmpxEqU32", Operand(0), 3), + }; + for (uint index = 1; index < 6; index++) + instructions.Add(Vop1(28 + index * 4, "VMovB32", 10 + index, Operand(BitConverter.SingleToUInt32Bits(1)))); + if (useAbsolutePatchId) instructions.Add(Vop1(52, "VCvtF32U32", 10, Gen5Operand.Vector(factorSourceRegister))); + var storePc = useAbsolutePatchId ? 56u : 52u; + var first = BufferAccess(storePc, "BufferStoreDwordx4", 24, dwords: 4, vectorData: 10); + var last = BufferAccess(storePc + 8, "BufferStoreDwordx2", 24, dwords: 2, vectorData: 14, offset: 16); + instructions.Add(first with { Sources = [..first.Sources.Take(2), Gen5Operand.Scalar(4)] }); + instructions.Add(last with { Sources = [..last.Sources.Take(2), Gen5Operand.Scalar(4)] }); + instructions.Add(EndProgram(storePc + 16)); + var program = new Gen5ShaderProgram(0x1000, instructions); + var plan = ShaderResourcePlan.Extract(program, ShaderStage.Compute, Hash, 8, 20, waveSize: 64); + var resources = ResourceMaterializer.ApplyTo(plan, ResourceSpecialization.Default(plan.Info)); + var layout = BindingLayout.Allocate(resources.Info, BindingLayout.CollectUserDataRegisters(program, 8, 20), false, false, false, usesTessellationData: true); + var request = new ShaderCompileRequest(plan, resources, layout) + { LocalSizeX = 64, LocalDataShareDwords = 64, WaveSize = 64, CooperativeWave64Workgroup = true, TessellationHull = new(4, 4, 1, 0) }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + var info = new ShaderProgramInfo + { + Stage = ShaderStageKind.Compute, Hash = Hash, UserDataBase = 8, UserDataCount = 20, + Bindings = layout, Resources = resources, TessellationFactorBuffer = (int)Gen5TessellationLowering.FindFactorBuffer(plan, 0), + Buffers = [new(false, true, false, false, false, 24, 0)], + }; + var data = new uint[20]; + data[12] = BitConverter.SingleToUInt32Bits(firstOuter); + data[16] = (uint)guestFactors; data[17] = (uint)(guestFactors >> 32); data[18] = factorRingBytes; + var input = new ComputeInputInfo + { + ThreadsX = 64, ThreadsY = 1, ThreadsZ = 1, WaveSize = 64, + Stage = new(info, new ResourceSnapshot { UserData = data, Buffers = [[(uint)guestFactors, (uint)(guestFactors >> 32), factorRingBytes, 0]] }), + }; + return new() { Input = input, Program = new(5, host.CreateShaderModule(new VulkanCompiledGuestShader(shader.Spirv), ShaderStage.Compute, Hash, 5)) }; + } + + public PipelineHandle CreateGraphicsPipeline(ReadOnlySpan colors, in DepthAttachmentState depth, + VertexInputInfo vertexInput, PixelInputInfo? pixelInput, ContextRegisters context, in RenderingState rendering, + PrimitiveTopology topology, bool restart, bool disableBlending, ShaderProgram vertexProgram, ShaderProgram pixelProgram) + { + var basic = ShaderPipelineCache.BuildGraphicsDescription(colors, depth, vertexInput, pixelInput, context, rendering, + PrimitiveTopology.PatchList, false, disableBlending, vertexProgram, pixelProgram, host.NoAttachmentSampleCounts); + return host.CreateGraphicsPipeline(new() + { + Rendering = basic.Rendering, VertexInput = basic.VertexInput, VertexInfo = basic.VertexInfo, + VertexProgram = basic.VertexProgram, VertexStage = basic.VertexStage, PixelInfo = basic.PixelInfo, + PixelProgram = basic.PixelProgram, PixelStage = basic.PixelStage, StaticParameters = basic.StaticParameters, + Tessellation = new(_control, _domain, 4), + }); + } + public ComputeProgram GetComputeProgram(ComputeStageRegisters compute, ShaderInterfaceRegisters shaderInterface, + uint initiator, uint x, uint y, uint z) => throw new NotSupportedException(); + public PipelineHandle CreateComputePipeline(ComputeInputInfo input, ShaderProgram program) => + host.CreateComputePipeline(new() { Input = input, Program = program, Stage = input.Stage.Program! }); + } +} diff --git a/tests/SharpEmu.Libs.Tests/VideoOut/RenderHostDeviceTests.cs b/tests/SharpEmu.Libs.Tests/VideoOut/RenderHostDeviceTests.cs index 1150002..331bba3 100644 --- a/tests/SharpEmu.Libs.Tests/VideoOut/RenderHostDeviceTests.cs +++ b/tests/SharpEmu.Libs.Tests/VideoOut/RenderHostDeviceTests.cs @@ -395,10 +395,15 @@ public sealed unsafe partial class RenderHostDeviceTests : IClassFixture> 32), 0x10000, 0]] }; presenter.Run(() => { + if (gpuOwned) image.MarkGpuModified(); Assert.False(image.IsBufferModified); using var preparation = presenter.RenderHost.BeginPreparation(); var prepared = presenter.RenderHost.PrepareBindings(new ShaderStageResources(program, snapshot)); presenter.RenderHost.BindResources(prepared); - Assert.Equal(formatted && writable, image.IsBufferModified); + Assert.Equal(writable && (formatted || !gpuOwned), image.IsBufferModified); + Assert.Equal(gpuOwned && !(formatted && writable), image.IsGpuModified); }); harness.Finish(); harness.Shutdown(); @@ -591,6 +598,54 @@ public sealed unsafe partial class RenderHostDeviceTests : IClassFixture harness.Images.TryClearImageFromBuffer(address, 0x8000, 0x11223344))); + Assert.True(harness.Image(imageId).IsGpuModified); + var read = new BufferResource { Read = true, MaxByteExtent = 4 }; + var write = new BufferResource { Written = true, Formatted = true, MaxByteExtent = 4 }; + var readAddress = address + 0x100; + var writeAddress = address + 0x4000; + uint[] Descriptor(ulong at) => [(uint)at, (uint)(at >> 32), 4, 0]; + + presenter.Run(() => + { + using var preparation = presenter.RenderHost.BeginPreparation(); + var firstProgram = FixedProgramProvider.EmptyProgram(ShaderStageKind.Compute, 3, + new ShaderResourceInfo { Buffers = separateStages ? [read] : [read, write] }); + var first = presenter.RenderHost.PrepareBindings(new ShaderStageResources(firstProgram, + new ResourceSnapshot { Buffers = separateStages ? [Descriptor(readAddress)] : [Descriptor(readAddress), Descriptor(writeAddress)] })); + IPreparedBindings? second = null; + if (separateStages) + { + var secondProgram = FixedProgramProvider.EmptyProgram(ShaderStageKind.Compute, 4, + new ShaderResourceInfo { Buffers = [write] }); + second = presenter.RenderHost.PrepareBindings(new ShaderStageResources(secondProgram, + new ResourceSnapshot { Buffers = [Descriptor(writeAddress)] })); + } + + var readOwner = harness.Cache.GetBuffer(harness.Cache.FindBuffer(readAddress, 4)); + var writeOwner = harness.Cache.GetBuffer(harness.Cache.FindBuffer(writeAddress, 4)); + presenter.RenderHost.BindResources(first); + if (second is not null) presenter.RenderHost.BindResources(second); + Assert.Same(readOwner, harness.Cache.GetBuffer(harness.Cache.FindBuffer(readAddress, 4))); + Assert.Same(writeOwner, harness.Cache.GetBuffer(harness.Cache.FindBuffer(writeAddress, 4))); + }); + harness.Shutdown(); + } + [Fact] public void PrepareBindings_PrivateBufferUploadsCurrentBytesAndStillRejectsGpuWrites() { @@ -966,6 +1021,8 @@ public sealed unsafe partial class RenderHostDeviceTests : IClassFixture arguments = stackalloc byte[3 * sizeof(uint)]; + Assert.False(presenter.RenderHost.TryReadCleanGuestBytes(unmappedAddress, arguments)); }); } @@ -984,6 +1041,7 @@ public sealed unsafe partial class RenderHostDeviceTests : IClassFixture arguments = stackalloc byte[3 * sizeof(uint)]; + Assert.True(renderHost.TryReadCleanGuestBytes(address, arguments)); + Assert.Equal(0xCAFEF00Du, BitConverter.ToUInt32(arguments)); + // CPU-owned bytes on a page that also holds GPU-owned bytes are read through the + // backing alias, without bringing the neighbouring GPU bytes back. + _ = harness.Cache.ObtainBuffer(address, 0x1000, isWritten: false); + _ = harness.Cache.ObtainBuffer(address + 0x800, 4, isWritten: true); + Assert.True(harness.Cache.HasGpuDirtyPages(address, 4)); + Assert.False(harness.Cache.HasGpuDirtyBytes(address, 4)); + Span resident = stackalloc byte[4]; + Assert.True(host.TryReadResidentGuestBytes(address, resident, clean: true)); + Assert.Equal(0xCAFEF00Du, System.Buffers.Binary.BinaryPrimitives.ReadUInt32LittleEndian(resident)); + Assert.True(harness.Cache.HasGpuDirtyBytes(address + 0x800, 4)); + + // A clean word read brings GPU-owned bytes back first; byte reads still refuse them. _ = harness.Cache.ObtainBuffer(address, 0x1000, isWritten: true); - Assert.False(host.TryReadCleanGuestWord(address, out _)); - Assert.False(host.TryReadCleanGuestWord(address + 0x800, out _)); + Assert.False(renderHost.TryReadCleanGuestBytes(address, arguments)); + Assert.True(host.TryReadCleanGuestWord(address, out word)); + Assert.Equal(0xCAFEF00Du, word); Assert.True(host.TryReadCleanGuestWord(address + 0x2000, out _)); Assert.True(host.TryReadGuestWord(address, out word)); Assert.Equal(0xCAFEF00Du, word); @@ -1011,6 +1085,70 @@ public sealed unsafe partial class RenderHostDeviceTests : IClassFixture (long)verifications.GetValue(presenter.Instance)!; + var request = SharpEmu.Libs.Tests.Gpu.Images.ImageCacheTestSupport.LinearRequest(address, 0x1000, Format.R8G8B8A8Unorm, + GuestPixelFormat.Bits8_8_8_8UNorm, GuestImageType.Color2D, new Extent3D(32, 32, 1), 1, 4, 1); + GuestGpuMemoryHook.Attach(harness.Gpu); + try + { + presenter.Run(() => + { + Span bytes = stackalloc byte[sizeof(uint)]; + Assert.True(renderHost.TryReadCleanGuestBytes(address, bytes)); + var first = Verifications(); + for (var index = 0; index < 16; index++) + { + Assert.True(renderHost.TryReadCleanGuestBytes(address + (ulong)index * 4, bytes)); + } + + var host = (IShaderPipelineHost)presenter.Instance; + for (var index = 0; index < 16; index++) + { + Assert.True(host.TryReadCleanGuestWord(address + (ulong)index * 4, out var word)); + if (index == 0) Assert.Equal(0xCAFEF00Du, word); + } + + Assert.Equal(first, Verifications()); + Assert.True(renderHost.TryReadCleanGuestBytes(address, bytes)); + Assert.Equal(0xCAFEF00Du, BitConverter.ToUInt32(bytes)); + }); + var image = harness.Acquire(ref request); + presenter.Run(() => + { + harness.Image(image).MarkGpuModified(); + var before = Verifications(); + Span bytes = stackalloc byte[sizeof(uint)]; + Assert.False(renderHost.TryReadCleanGuestBytes(address, bytes)); + Assert.True(Verifications() > before); + }); + } + finally + { + GuestGpuMemoryHook.Attach(null); + } + + harness.Finish(); + harness.Shutdown(); + } + [Fact] public void ConsecutiveDraws_ShareOneRenderingScopeAndCloseItBeforeTheSubmit() { diff --git a/tests/SharpEmu.Libs.Tests/VideoOut/ShaderPrewarmPresenterTests.cs b/tests/SharpEmu.Libs.Tests/VideoOut/ShaderPrewarmPresenterTests.cs index b3de976..327a051 100644 --- a/tests/SharpEmu.Libs.Tests/VideoOut/ShaderPrewarmPresenterTests.cs +++ b/tests/SharpEmu.Libs.Tests/VideoOut/ShaderPrewarmPresenterTests.cs @@ -46,6 +46,9 @@ public sealed class ShaderPrewarmPresenterTests(HeadlessVulkanFixture fixture) : using (var list = ShaderPrewarmList.Open(_directory)!) { var guest = new PipelineTestGuest(Compile); + // Match the presenter's runtime buffer metadata policy before comparing + // module identities; formatted bounds also consume the packed tail size. + guest.Host.RuntimeBufferStridesEnabled = true; guest.Host.ShaderPrewarm = list; guest.RegisterProgram(CodeAddress, HeaderAddress, PipelineTestGuest.FormatLoadProgram); var cursor = 0u; @@ -62,7 +65,8 @@ public sealed class ShaderPrewarmPresenterTests(HeadlessVulkanFixture fixture) : var prewarmed = new ConcurrentDictionary(StringComparer.Ordinal); presenter.SetField("_prewarmedShaderIdentities", prewarmed); - presenter.InvokeMethod("PrewarmComputePipeline", record, code, new FakeShaderCompiler(Compile), false, true); + presenter.InvokeMethod("PrewarmComputePipeline", record, code, new FakeShaderCompiler(Compile), + (IShaderPipelineHost)presenter.Instance); Assert.Equal(0, presenter.GetField("_shaderPrewarmFailed")); Assert.Equal(1, presenter.GetField("_shaderPrewarmCompiled")); diff --git a/tests/SharpEmu.Libs.Tests/VideoOut/VideoOutFlipRequestTests.cs b/tests/SharpEmu.Libs.Tests/VideoOut/VideoOutFlipRequestTests.cs index 3cb5b8a..c2c9fc5 100644 --- a/tests/SharpEmu.Libs.Tests/VideoOut/VideoOutFlipRequestTests.cs +++ b/tests/SharpEmu.Libs.Tests/VideoOut/VideoOutFlipRequestTests.cs @@ -414,6 +414,21 @@ public sealed class VideoOutFlipRequestTests : IDisposable Assert.Null(VideoOutExports.GetFlipOutcomeForTests(request)); } + // A CPU flip reserved before its preparation reaches the graphics queue must not hold back an earlier wait. + [Fact] + public void CpuFlip_BlocksBufferWaitsOnlyOnceTheGraphicsQueueReachesIt() + { + Assert.Equal(0, VideoOutExports.TryReserveFlipRequest(_handle, -1, 0, 13, gpuQueued: false, out var request)); + Assert.True(VideoOutExports.IsFlipDone(_handle, -1)); + + VideoOutExports.MarkFlipOrdered(request); + Assert.False(VideoOutExports.IsFlipDone(_handle, -1)); + + VideoOutExports.CompleteFlip(request); + VideoOutExports.MarkFlipPresented(request); + Assert.True(VideoOutExports.IsFlipDone(_handle, -1)); + } + // Update completion counters even if the presenter has already finished the frame. [Fact] public void PresentThenComplete_StillCountsTheFlip() diff --git a/tests/SharpEmu.Libs.Tests/VideoOut/VideoOutOutputSupportTests.cs b/tests/SharpEmu.Libs.Tests/VideoOut/VideoOutOutputSupportTests.cs index cbf2e08..34e8d30 100644 --- a/tests/SharpEmu.Libs.Tests/VideoOut/VideoOutOutputSupportTests.cs +++ b/tests/SharpEmu.Libs.Tests/VideoOut/VideoOutOutputSupportTests.cs @@ -48,7 +48,7 @@ public sealed class VideoOutOutputSupportTests Assert.Equal(expectedHdr, supported); var bytes = Enumerable.Repeat((byte)0xA5, 0x32).ToArray(); - VideoOutExports.WriteOutputStatus(bytes.AsSpan(1, 0x30), 3840, 2160, 120, supported); + VideoOutExports.WriteOutputStatus(bytes.AsSpan(1, 0x30), GuestDisplayResolution.UltraHd, 120, supported); Assert.Equal(0xA5, bytes[0]); Assert.Equal(0xA5, bytes[^1]); diff --git a/tests/SharpEmu.Libs.Tests/VideoOut/VulkanPresentEncodeFormatTests.cs b/tests/SharpEmu.Libs.Tests/VideoOut/VulkanPresentEncodeFormatTests.cs index d47c7b3..0d994c2 100644 --- a/tests/SharpEmu.Libs.Tests/VideoOut/VulkanPresentEncodeFormatTests.cs +++ b/tests/SharpEmu.Libs.Tests/VideoOut/VulkanPresentEncodeFormatTests.cs @@ -9,6 +9,18 @@ namespace SharpEmu.Libs.Tests.VideoOut; public sealed class VulkanPresentEncodeFormatTests { + // Ghost of Yotei binds an 11_11_10 target with the UNORM number type; the format holds + // unsigned floats whatever the number type, as the image path already maps it. + [Theory] + [InlineData(0u)] + [InlineData(7u)] + public void Packed11_11_10Target_UsesTheUnsignedFloatFormat(uint numberType) + { + Assert.True(VulkanVideoPresenter.TryDecodeRenderTargetFormat(6, numberType, 0, out var decoded)); + Assert.Equal(Format.B10G11R11UfloatPack32, decoded.Format); + Assert.Equal(SharpEmu.ShaderCompiler.Gen5PixelOutputKind.Float, decoded.OutputKind); + } + [Fact] public void UnsignedByteTargetUsesUnsignedPixelOutput() { @@ -160,6 +172,20 @@ public sealed class VulkanPresentEncodeFormatTests Assert.Equal(expectedMapping, decoded.ExportMapping.Packed); } + [Theory] + [InlineData(1u, 6u, Format.R8Srgb)] + [InlineData(1u, 9u, Format.R8Srgb)] + [InlineData(3u, 6u, Format.R8G8Srgb)] + [InlineData(3u, 9u, Format.R8G8Srgb)] + public void TryDecodeRenderTargetFormat_DecodesNarrowSrgbOutputs(uint layout, uint numberType, Format expected) + { + Assert.True(VulkanVideoPresenter.TryDecodeRenderTargetFormat(layout, numberType, 0, out var decoded)); + Assert.Equal(expected, decoded.Format); + Assert.Equal(SharpEmu.ShaderCompiler.Gen5PixelOutputKind.Float, decoded.OutputKind); + Assert.True(decoded.ExportMapping.IsIdentity); + Assert.False(VulkanVideoPresenter.TryDecodeRenderTargetFormat(layout, numberType, 4, out _)); + } + [Theory] [InlineData(6u, 0u, Format.R8G8B8A8Srgb)] [InlineData(6u, 1u, Format.B8G8R8A8Srgb)] diff --git a/tests/SharpEmu.ShaderCompiler.Tests/DeviceAddressCodeSizeTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/DeviceAddressCodeSizeTests.cs new file mode 100644 index 0000000..d2178b3 --- /dev/null +++ b/tests/SharpEmu.ShaderCompiler.Tests/DeviceAddressCodeSizeTests.cs @@ -0,0 +1,92 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.ShaderCompiler; +using SharpEmu.ShaderCompiler.Vulkan; +using System.Buffers.Binary; +using Xunit; +using static SharpEmu.ShaderCompiler.Tests.Resources.ResourceTestProgram; + +namespace SharpEmu.ShaderCompiler.Tests; + +public sealed class DeviceAddressCodeSizeTests +{ + [Theory] + [InlineData(1)] + [InlineData(2)] + [InlineData(3)] + public void PartialDeviceFormatLoad_ConvertsOnlyItsRequestedComponents(uint count) + { + var request = Request(Program(ReadFirstLane(0, 12, 0), MoveScalar(4, 13, 0), + MoveScalar(8, 14, 16), MoveScalar(12, 15, 0), + ScalarBufferLoad(16, 12, destination: 20, count: 4), + BufferLoad(24, 20, dwords: count, formatted: true), EndProgram(32))); + Assert.Contains(request.Memory.Entries, memory => memory.Pc == 24 && memory.DeviceDescriptor); + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + Gen5LargeDispatcherValidationTests.ValidateWithSpirvToolsWhenAvailable(shader.Spirv); + var divisions = 0; + for (var offset = 20; offset < shader.Spirv.Length;) + { + var header = BinaryPrimitives.ReadUInt32LittleEndian(shader.Spirv.AsSpan(offset)); + if ((header & 0xFFFF) == (uint)SpirvOp.FDiv) divisions++; + offset += checked((int)(header >> 16) * 4); + } + Assert.Equal((int)count * 2, divisions); + } + [Theory] + [InlineData(1)] + [InlineData(2)] + [InlineData(4)] + public void RepeatedDeviceFormatLoads_ShareFormatDecoding(uint components) + { + byte[] Compile(int count) + { + var instructions = new List + { + ReadFirstLane(0, 12, 0), MoveScalar(4, 13, 0), + MoveScalar(8, 14, 16), MoveScalar(12, 15, 0), + ScalarBufferLoad(16, 12, destination: 20, count: 4), + }; + for (uint index = 0; index < count; index++) + instructions.Add(BufferLoad(24 + index * 8, 20, + offset: (int)index * 16, dwords: components, formatted: true)); + instructions.Add(EndProgram(24 + (uint)count * 8)); + var request = Request(Program(instructions.ToArray())); + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + Gen5LargeDispatcherValidationTests.ValidateWithSpirvToolsWhenAvailable(shader.Spirv); + return shader.Spirv; + } + + var single = Compile(1); + var repeated = Compile(16); + // Addresses and destination writes vary at every site. The runtime format + // catalogue, conversions, swizzles and page walks must not be duplicated. + Assert.True(repeated.Length - single.Length < 15 * 1500, + $"Additional device formatted loads grew SPIR-V by {repeated.Length - single.Length} bytes."); + } + + [Fact] + public void RepeatedRuntimeFormats_DoNotDuplicateTheLayoutCatalogue() + { + byte[] Compile(int count) + { + var instructions = new List(); + for (uint index = 0; index < count; index++) + instructions.Add(BufferAccess(index * 8, "BufferLoadFormatXyzw", 8, + (int)index * 16, 4, vectorData: 4)); + instructions.Add(EndProgram((uint)count * 8)); + var (plan, resources, layout) = Prepare(Program(instructions.ToArray())); + var request = new ShaderCompileRequest(plan, resources, layout) + { + LocalSizeX = 1, ThreadCountX = 1, ForceGenericBufferFormats = true, + }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + Gen5LargeDispatcherValidationTests.ValidateWithSpirvToolsWhenAvailable(shader.Spirv); + return shader.Spirv; + } + var single = Compile(1); + var repeated = Compile(16); + Assert.True(repeated.Length - single.Length < 15 * 18000, + $"Additional formatted loads grew SPIR-V by {repeated.Length - single.Length} bytes."); + } +} diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Gen5BarrierMemoryScopeTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Gen5BarrierMemoryScopeTests.cs new file mode 100644 index 0000000..eb1f718 --- /dev/null +++ b/tests/SharpEmu.ShaderCompiler.Tests/Gen5BarrierMemoryScopeTests.cs @@ -0,0 +1,149 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Buffers.Binary; +using SharpEmu.HLE; +using SharpEmu.ShaderCompiler.Tests.Resources; +using SharpEmu.ShaderCompiler.Vulkan; +using Xunit; + +namespace SharpEmu.ShaderCompiler.Tests; + +// What s_barrier publishes is whatever the guest waited for before it. +public sealed class Gen5BarrierMemoryScopeTests +{ + private const ulong ShaderAddress = 0x1_0000_0000; + private const uint Barrier = 0xBF8A0000; + private const uint EndProgram = 0xBF810000; + private const uint WaitLgkmOnly = 0xBF8CC07F; // s_waitcnt lgkmcnt(0) + private const uint WaitVectorMemory = 0xBF8C0F70; // s_waitcnt vmcnt(0) + + private const uint AcquireRelease = 0x8; + private const uint UniformMemory = 0x40; + private const uint WorkgroupMemory = 0x100; + private const uint ImageMemory = 0x800; + + [Theory] + [InlineData(WaitLgkmOnly, AcquireRelease | WorkgroupMemory)] + [InlineData(WaitVectorMemory, AcquireRelease | UniformMemory | WorkgroupMemory | ImageMemory)] + public void BarrierPublishesWhatTheWaitCountWaitedFor(uint wait, uint expected) + { + var program = Decode([wait, Barrier, EndProgram]); + Assert.Equal("SBarrier", program.Instructions[1].Opcode); + + var request = ResourceTestProgram.Request(program, userDataCount: 0); + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + Assert.Equal(expected, ControlBarrierSemantics(shader.Spirv)); + } + + private const uint WaitStoresZeroSopk = 0xBBFD0000; // s_waitcnt_vscnt null, 0 + private const uint WaitLgkmZeroSopk = 0xBD7D0000; // s_waitcnt_lgkmcnt null, 0 + + // GFX10 tracks stores with vscnt, and a drained counter stays drained for the block. + [Theory] + [InlineData(new[] { WaitStoresZeroSopk, WaitLgkmOnly }, AcquireRelease | UniformMemory | WorkgroupMemory | ImageMemory)] + [InlineData(new[] { WaitVectorMemory, WaitLgkmOnly }, AcquireRelease | UniformMemory | WorkgroupMemory | ImageMemory)] + [InlineData(new[] { WaitLgkmZeroSopk }, AcquireRelease | WorkgroupMemory)] + public void StoreWaitsCountAndStayCountedAcrossLaterWaits(uint[] waits, uint expected) + { + var program = Decode([.. waits, Barrier, EndProgram]); + var request = ResourceTestProgram.Request(program, userDataCount: 0); + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + Assert.Equal(expected, ControlBarrierSemantics(shader.Spirv)); + } + + // A barrier with no wait before it in its block keeps the conservative semantics: the + // publish may have happened in a block the translator cannot order against this one. + [Fact] + public void BarrierWithoutAWaitCountKeepsDeviceMemoryOrdering() + { + var program = Decode([Barrier, EndProgram]); + var request = ResourceTestProgram.Request(program, userDataCount: 0); + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + Assert.Equal( + AcquireRelease | UniformMemory | WorkgroupMemory | ImageMemory, + ControlBarrierSemantics(shader.Spirv)); + } + + // The value of the memory-semantics operand of the module's single OpControlBarrier. + private static uint ControlBarrierSemantics(byte[] spirv) + { + var words = new uint[spirv.Length / 4]; + Buffer.BlockCopy(spirv, 0, words, 0, spirv.Length); + var constants = new Dictionary(); + uint? semanticsId = null; + for (var offset = 5; offset < words.Length;) + { + var wordCount = (int)(words[offset] >> 16); + var opcode = (ushort)(words[offset] & 0xFFFF); + if (opcode == (ushort)SpirvOp.Constant && wordCount == 4) + { + constants[words[offset + 2]] = words[offset + 3]; + } + else if (opcode == (ushort)SpirvOp.ControlBarrier) + { + Assert.Null(semanticsId); + semanticsId = words[offset + 3]; + } + + offset += wordCount; + } + + Assert.NotNull(semanticsId); + return constants[semanticsId.Value]; + } + + private static Gen5ShaderProgram Decode(IReadOnlyList words) + { + var memory = new TestCpuMemory(ShaderAddress, words.Count * sizeof(uint)); + var bytes = new byte[words.Count * sizeof(uint)]; + for (var index = 0; index < words.Count; index++) + { + BinaryPrimitives.WriteUInt32LittleEndian(bytes.AsSpan(index * sizeof(uint)), words[index]); + } + + Assert.True(memory.TryWrite(ShaderAddress, bytes)); + var context = new CpuContext(memory, Generation.Gen5); + Assert.True(Gen5ShaderTranslator.TryDecodeProgram(context, ShaderAddress, out var program, out var error), error); + return program; + } + + private sealed class TestCpuMemory(ulong baseAddress, int size) : ICpuMemory + { + private readonly byte[] _storage = new byte[size]; + + public bool TryRead(ulong virtualAddress, Span destination) + { + if (!TryResolve(virtualAddress, destination.Length, out var offset)) + { + return false; + } + + _storage.AsSpan(offset, destination.Length).CopyTo(destination); + return true; + } + + public bool TryWrite(ulong virtualAddress, ReadOnlySpan source) + { + if (!TryResolve(virtualAddress, source.Length, out var offset)) + { + return false; + } + + source.CopyTo(_storage.AsSpan(offset, source.Length)); + return true; + } + + private bool TryResolve(ulong address, int length, out int offset) + { + offset = 0; + if (address < baseAddress || address - baseAddress > int.MaxValue) + { + return false; + } + + offset = (int)(address - baseAddress); + return offset <= _storage.Length - length; + } + } +} diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Gen5DualSourceBlendTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Gen5DualSourceBlendTests.cs new file mode 100644 index 0000000..f7966dc --- /dev/null +++ b/tests/SharpEmu.ShaderCompiler.Tests/Gen5DualSourceBlendTests.cs @@ -0,0 +1,70 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Buffers.Binary; +using SharpEmu.ShaderCompiler; +using SharpEmu.ShaderCompiler.Resources; +using SharpEmu.ShaderCompiler.Tests.Resources; +using SharpEmu.ShaderCompiler.Vulkan; +using Xunit; + +namespace SharpEmu.ShaderCompiler.Tests; + +// Dual-source blending reads a second color from the MRT1 export. It is declared at target +// 0's location with index 1 and is not a render target of its own. +public sealed class Gen5DualSourceBlendTests +{ + private static Gen5ShaderInstruction Export(uint pc, uint target, bool done) => + new(pc, Gen5ShaderEncoding.Exp, "Exp", [0u, 0u], + [Gen5Operand.Vector(0), Gen5Operand.Vector(1), Gen5Operand.Vector(2), Gen5Operand.Vector(3)], [], + new Gen5ExportControl(target, 15, false, done, false)); + + private static ShaderCompileRequest Request(params Gen5PixelOutputBinding[] outputs) + { + var program = ResourceTestProgram.Program(Export(0, 0, false), Export(8, 1, true), ResourceTestProgram.EndProgram(16)); + var (plan, resources, layout) = ResourceTestProgram.Prepare(program, ShaderStage.Pixel, userDataCount: 0); + return new ShaderCompileRequest(plan, resources, layout) { PixelOutputs = outputs }; + } + + private static IEnumerable<(uint Op, uint[] Args)> Instructions(byte[] bytes) + { + for (var offset = 20; offset < bytes.Length;) + { + var header = BinaryPrimitives.ReadUInt32LittleEndian(bytes.AsSpan(offset)); + var count = (int)(header >> 16); + var args = new uint[count - 1]; + for (var index = 0; index < args.Length; index++) + args[index] = BinaryPrimitives.ReadUInt32LittleEndian(bytes.AsSpan(offset + 4 * (index + 1))); + yield return (header & 0xFFFF, args); + offset += count * 4; + } + } + + [Fact] + public void SecondSource_IsLocationZeroIndexOne() + { + var request = Request( + new Gen5PixelOutputBinding(0, 0, Gen5PixelOutputKind.Float), + new Gen5PixelOutputBinding(1, 0, Gen5PixelOutputKind.Float) { ExportTarget = 1, Index = 1 }); + + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + Gen5LargeDispatcherValidationTests.ValidateWithSpirvToolsWhenAvailable(shader.Spirv); + const uint Decorate = 71, Location = 30, Index = 32; + var decorations = Instructions(shader.Spirv).Where(i => i.Op == Decorate).ToArray(); + var second = Assert.Single(decorations, i => i.Args[1] == Index && i.Args[2] == 1).Args[0]; + Assert.Contains(decorations, i => i.Args[0] == second && i.Args[1] == Location && i.Args[2] == 0); + Assert.Equal(2, decorations.Count(i => i.Args[1] == Location && i.Args[2] == 0)); + } + + [Fact] + public void SecondSource_NeedsASingleTarget() + { + var request = Request( + new Gen5PixelOutputBinding(0, 0, Gen5PixelOutputKind.Float), + new Gen5PixelOutputBinding(2, 1, Gen5PixelOutputKind.Float), + new Gen5PixelOutputBinding(1, 0, Gen5PixelOutputKind.Float) { ExportTarget = 1, Index = 1 }); + + Assert.False(Gen5SpirvTranslator.TryCompileProgram(request, out _, out var error)); + Assert.Contains("second source", error, StringComparison.Ordinal); + } +} diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Gen5DumpedSkeletonTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Gen5DumpedSkeletonTests.cs new file mode 100644 index 0000000..a76db67 --- /dev/null +++ b/tests/SharpEmu.ShaderCompiler.Tests/Gen5DumpedSkeletonTests.cs @@ -0,0 +1,76 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Globalization; +using SharpEmu.ShaderCompiler.Vulkan; +using Xunit; +using Xunit.Abstractions; +using static SharpEmu.ShaderCompiler.Tests.Resources.ResourceTestProgram; + +namespace SharpEmu.ShaderCompiler.Tests; + +// Replays the branch skeleton of every dumped program (`*.input.ir.txt`) in the directory named +// by SHARPEMU_CFG_SKELETON_DIR through the structured planner. Every other instruction becomes a +// no-op, so only the control flow is tested. Without the variable the test does nothing. +public sealed class Gen5DumpedSkeletonTests(ITestOutputHelper output) +{ + [Fact] + public void DumpedSkeletonsAreStructured() + { + var directory = Environment.GetEnvironmentVariable("SHARPEMU_CFG_SKELETON_DIR"); + if (string.IsNullOrEmpty(directory)) + { + return; + } + + var failures = new List(); + foreach (var file in Directory.EnumerateFiles(directory, "*.input.ir.txt")) + { + var program = Program([.. ReadSkeleton(file)]); + var code = Gen5StructuredControlFlowTests.Compile(program, forceDispatcher: false); + if (!Gen5LargeDispatcherValidationTests.ReadSpirvOpcodes(code).Contains(SpirvOp.FunctionCall)) + { + failures.Add(Path.GetFileName(file)); + continue; + } + + Gen5LargeDispatcherValidationTests.ValidateStructuredControlFlow(code); + Gen5LargeDispatcherValidationTests.ValidateWithSpirvToolsWhenAvailable(code); + } + + foreach (var failure in failures) + { + output.WriteLine(failure); + } + + Assert.Empty(failures); + } + + private static IEnumerable ReadSkeleton(string file) + { + foreach (var line in File.ReadLines(file)) + { + var fields = line.Split(' ', StringSplitOptions.RemoveEmptyEntries); + if (fields.Length < 3 || !fields[0].StartsWith("0x", StringComparison.Ordinal)) + { + continue; + } + + var pc = uint.Parse(fields[0].AsSpan(2), NumberStyles.HexNumber); + var word = uint.Parse(fields[1].Split('_')[0], NumberStyles.HexNumber); + var opcode = fields[2]; + if (opcode == "SEndpgm") + { + yield return EndProgram(pc); + } + else if (opcode == "SBranch" || opcode.StartsWith("SCbranch", StringComparison.Ordinal)) + { + yield return Branch(pc, opcode, unchecked((short)(word & 0xFFFF))); + } + else + { + yield return Nop(pc); + } + } + } +} diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Gen5ExecFullAnalysisTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Gen5ExecFullAnalysisTests.cs index caffd90..5b6f0fd 100644 --- a/tests/SharpEmu.ShaderCompiler.Tests/Gen5ExecFullAnalysisTests.cs +++ b/tests/SharpEmu.ShaderCompiler.Tests/Gen5ExecFullAnalysisTests.cs @@ -14,10 +14,11 @@ public sealed class Gen5ExecFullAnalysisTests private readonly List _program = []; private uint _pc; - private uint Add(string opcode, Gen5ShaderEncoding encoding, Gen5Operand[] destinations, Gen5Operand[] sources, uint word = 0) + private uint Add(string opcode, Gen5ShaderEncoding encoding, Gen5Operand[] destinations, Gen5Operand[] sources, + uint word = 0, Gen5InstructionControl? control = null) { var pc = _pc; - _program.Add(new Gen5ShaderInstruction(pc, encoding, opcode, [word], sources, destinations, null)); + _program.Add(new Gen5ShaderInstruction(pc, encoding, opcode, [word], sources, destinations, control)); _pc += 4; return pc; } @@ -66,6 +67,57 @@ public sealed class Gen5ExecFullAnalysisTests Assert.DoesNotContain(after, full); } + [Theory] + [InlineData(false, false, 40u)] + [InlineData(false, false, 106u)] + [InlineData(false, true, 40u)] + [InlineData(false, true, 106u)] + [InlineData(true, false, 40u)] + [InlineData(true, false, 106u)] + [InlineData(true, true, 40u)] + [InlineData(true, true, 106u)] + public void CompareIntoExecWithLegacyDestination_PreservesSavedMask(bool wave32, bool useVop3, uint savedMask) + { + Scalar("SMovB64", savedMask, Gen5Operand.Scalar(Exec)); + var before = VectorAdd(); + Add("VCmpxNltF32", useVop3 ? Gen5ShaderEncoding.Vop3 : Gen5ShaderEncoding.Vopc, [], + [Gen5Operand.Vector(20), new Gen5Operand(Gen5OperandKind.EncodedConstant, 244)], + control: CompareControl(useVop3, savedMask)); + var masked = VectorAdd(); + Scalar("SMovB64", Exec, Gen5Operand.Scalar(savedMask)); + var restored = VectorAdd(); + + var full = Gen5ExecFullAnalysis.Analyze(new Gen5ShaderProgram(0x1000, _program), wave32); + Assert.Contains(before, full); + Assert.DoesNotContain(masked, full); + Assert.Contains(restored, full); + } + + [Theory] + [InlineData(false, false)] + [InlineData(false, true)] + [InlineData(true, false)] + [InlineData(true, true)] + public void CompareIntoScalarDestination_InvalidatesSavedMask(bool wave32, bool useVop3) + { + Scalar("SMovB64", 106, Gen5Operand.Scalar(Exec)); + Add("VCmpNltF32", useVop3 ? Gen5ShaderEncoding.Vop3 : Gen5ShaderEncoding.Vopc, [], + [Gen5Operand.Vector(20), new Gen5Operand(Gen5OperandKind.EncodedConstant, 244)], + control: CompareControl(useVop3, 106)); + var beforeRestore = VectorAdd(); + Scalar("SMovB64", Exec, Gen5Operand.Scalar(106)); + var afterRestore = VectorAdd(); + + var full = Gen5ExecFullAnalysis.Analyze(new Gen5ShaderProgram(0x1000, _program), wave32); + Assert.Contains(beforeRestore, full); + Assert.DoesNotContain(afterRestore, full); + } + + private static Gen5InstructionControl CompareControl(bool useVop3, uint scalarDestination) => + useVop3 + ? new Gen5Vop3Control(0, 0, 0, false, 0, scalarDestination) + : new Gen5SdwaControl(6, 0, 6, 6, false, false, 0, 0, 0, false, scalarDestination); + [Fact] public void AfterAnSdwaVectorCompareIntoExec_ExecIsUnknown() { diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Gen5ExecRunTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Gen5ExecRunTests.cs new file mode 100644 index 0000000..e6e58c8 --- /dev/null +++ b/tests/SharpEmu.ShaderCompiler.Tests/Gen5ExecRunTests.cs @@ -0,0 +1,201 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Buffers.Binary; +using SharpEmu.HLE; +using SharpEmu.ShaderCompiler.Ir; +using SharpEmu.ShaderCompiler.Tests.Resources; +using SharpEmu.ShaderCompiler.Vulkan; +using Xunit; + +namespace SharpEmu.ShaderCompiler.Tests; + +// A run of plain vector ALU writes may share one EXEC test. Anything the wave runs even where this +// lane is masked off has to end the run. +public sealed class Gen5ExecRunTests +{ + private const ulong ShaderAddress = 0x1_0000_0000; + private const uint EndProgram = 0xBF810000; + private const uint MultiplyF32 = 0x10040510; // v_mul_f32 v2, v16, v2 + private const uint CompareExecNeU32 = 0x7DAA0480; // v_cmpx_ne_u32 0, v2 (writes EXEC) + private const uint ConvertF32U32 = 0x7E1C0C10; // v_cvt_f32_u32 v14, s16 + private const uint ScalarMove = 0xBE8E047E; // s_mov_b64 s14, exec + + private static Gen5ShaderInstruction Only(uint word) + { + var program = Decode([word, EndProgram]); + return program.Instructions[0]; + } + + [Fact] + public void PlainVectorArithmeticCanShareAnExecTest() + { + Assert.True(Gen5ExecRunAnalysis.IsExecMaskedVectorAlu(Only(MultiplyF32))); + Assert.True(Gen5ExecRunAnalysis.IsExecMaskedVectorAlu(Only(ConvertF32U32))); + } + + // v_cmpx writes EXEC and v_cmp writes a mask register; GFX10 keeps that destination in the + // instruction's control, so the destination list alone does not show it. + [Fact] + public void ACompareThatWritesAMaskRegisterEndsTheRun() + { + var compare = Only(CompareExecNeU32); + Assert.Empty(compare.Destinations); + Assert.False(Gen5ExecRunAnalysis.IsExecMaskedVectorAlu(compare)); + } + + // SALU is not EXEC masked on the hardware and every invocation keeps its own scalar registers. + [Fact] + public void AScalarWriteEndsTheRun() => + Assert.False(Gen5ExecRunAnalysis.IsExecMaskedVectorAlu(Only(ScalarMove))); + + // The carry forms write VCC without naming it as a destination. + [Theory] + [InlineData("VAddCoU32")] + [InlineData("VAddCoCiU32")] + [InlineData("VSubCoU32")] + [InlineData("VSubrevCoCiU32")] + public void ACarryWritingAddEndsTheRun(string opcode) + { + var instruction = new Gen5ShaderInstruction( + 0, Gen5ShaderEncoding.Vop2, opcode, [0], [Gen5Operand.Vector(2), Gen5Operand.Vector(3)], + [Gen5Operand.Vector(5)], null); + Assert.False(Gen5ExecRunAnalysis.IsExecMaskedVectorAlu(instruction)); + } + + // A VOP3 compare names its mask register in the control, not in the destination list. + [Fact] + public void AVop3CompareDestinationInTheControlEndsTheRun() + { + var instruction = new Gen5ShaderInstruction( + 0, Gen5ShaderEncoding.Vop3, "VMadU64U32", [0], [Gen5Operand.Vector(2)], [Gen5Operand.Vector(4)], + new Gen5Vop3Control(0, 0, 0, false, 0, 20)); + Assert.False(Gen5ExecRunAnalysis.IsExecMaskedVectorAlu(instruction)); + + var plain = instruction with { Control = new Gen5Vop3Control(0, 0, 0, false, 0, null) }; + Assert.True(Gen5ExecRunAnalysis.IsExecMaskedVectorAlu(plain)); + } + + [Fact] + public void SharedMemoryEndsTheRun() + { + var instruction = new Gen5ShaderInstruction( + 0, Gen5ShaderEncoding.Ds, "VMovB32", [0], [Gen5Operand.Vector(2)], [Gen5Operand.Vector(1)], + new Gen5DataShareControl(0, 0, false)); + Assert.False(Gen5ExecRunAnalysis.IsExecMaskedVectorAlu(instruction)); + } + + // Cross-lane reads need the lanes they read from to have run. + [Theory] + [InlineData("VReadlaneB32")] + [InlineData("VWritelaneB32")] + [InlineData("VReadfirstlaneB32")] + [InlineData("VPermlane16B32")] + public void CrossLaneReadsEndTheRun(string opcode) + { + var instruction = new Gen5ShaderInstruction( + 0, Gen5ShaderEncoding.Vop1, opcode, [0], [Gen5Operand.Vector(1)], [Gen5Operand.Vector(2)], null); + Assert.False(Gen5ExecRunAnalysis.IsExecMaskedVectorAlu(instruction)); + } + + // A neighbour's register is only there if the neighbour ran. + [Fact] + public void TheDppEncodingsEndTheRun() + { + var dpp = new Gen5ShaderInstruction( + 0, Gen5ShaderEncoding.Vop1, "VMovB32", [0], [Gen5Operand.Vector(1)], [Gen5Operand.Vector(2)], + new Gen5Dpp8Control(0, false)); + Assert.False(Gen5ExecRunAnalysis.IsExecMaskedVectorAlu(dpp)); + } + + // A run under an EXEC the translator cannot prove full keeps one test for the whole run: the + // guarded translation selects against the old value of every register it writes instead. + [Fact] + public void ARunUnderAnUnknownExecKeepsOneTestInsteadOfOneSelectPerWrite() + { + var words = new List { CompareExecNeU32 }; + for (var index = 0; index < 16; index++) + { + words.Add(MultiplyF32); + } + + words.Add(EndProgram); + var program = Decode(words); + var request = ResourceTestProgram.Request(program, userDataCount: 0); + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + + // One shared test instead of one select per write: the sixteen multiplies contribute no + // selects of their own, so the module holds far fewer than sixteen. + Assert.True( + CountOpcode(shader.Spirv, SelectOpcode) < 16, + $"the run still selects per write: selects={CountOpcode(shader.Spirv, SelectOpcode)}"); + } + + private const uint SelectOpcode = 169; + + private static int CountOpcode(byte[] spirv, uint opcode) + { + var words = new uint[spirv.Length / 4]; + Buffer.BlockCopy(spirv, 0, words, 0, spirv.Length); + var count = 0; + for (var index = 5; index < words.Length;) + { + var length = words[index] >> 16; + if (length == 0) + { + break; + } + + if ((words[index] & 0xFFFF) == opcode) + { + count++; + } + + index += (int)length; + } + + return count; + } + + private sealed class TestCpuMemory(ulong baseAddress, int size) : ICpuMemory + { + private readonly byte[] _storage = new byte[size]; + + public bool TryRead(ulong address, Span destination) + { + if (address < baseAddress || address + (ulong)destination.Length > baseAddress + (ulong)size) + { + return false; + } + + _storage.AsSpan((int)(address - baseAddress), destination.Length).CopyTo(destination); + return true; + } + + public bool TryWrite(ulong address, ReadOnlySpan source) + { + if (address < baseAddress || address + (ulong)source.Length > baseAddress + (ulong)size) + { + return false; + } + + source.CopyTo(_storage.AsSpan((int)(address - baseAddress), source.Length)); + return true; + } + } + + private static Gen5ShaderProgram Decode(IReadOnlyList words) + { + var memory = new TestCpuMemory(ShaderAddress, words.Count * sizeof(uint)); + var bytes = new byte[words.Count * sizeof(uint)]; + for (var index = 0; index < words.Count; index++) + { + BinaryPrimitives.WriteUInt32LittleEndian(bytes.AsSpan(index * sizeof(uint)), words[index]); + } + + Assert.True(memory.TryWrite(ShaderAddress, bytes)); + var context = new CpuContext(memory, Generation.Gen5); + Assert.True(Gen5ShaderTranslator.TryDecodeProgram(context, ShaderAddress, out var program, out var error), error); + return program; + } +} diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Gen5Float16ArithmeticTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Gen5Float16ArithmeticTests.cs index 7d4fb4a..eebc600 100644 --- a/tests/SharpEmu.ShaderCompiler.Tests/Gen5Float16ArithmeticTests.cs +++ b/tests/SharpEmu.ShaderCompiler.Tests/Gen5Float16ArithmeticTests.cs @@ -14,6 +14,57 @@ public sealed class Gen5Float16ArithmeticTests private const ulong ShaderAddress = 0x1_0000_0000; private const uint SEndpgm = 0xBF810000; + [Theory] + [InlineData(false)] + [InlineData(true)] + public void PackedTruncation_PointerMatchesRegisterStorage(bool elideExecGuards) + { + var program = Decode([0x5E101B0Du, SEndpgm]); // v_cvt_pkrtz_f16_f32 v8, v13, v13 + var (plan, resources, bindings) = ResourceTestProgram.Prepare(program, userDataCount: 0); + var request = new ShaderCompileRequest(plan, resources, bindings) + { EnableExecGuardElision = elideExecGuards }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + // The packed-half shadow of v8 is one variable; its pointer type must carry the storage + // class it is declared with (Function when the register files live in the function). + var pointers = new Dictionary(); + var vec2Types = new HashSet(); + var packedHalfVariables = new List<(uint Type, uint Storage)>(); + for (var offset = 20; offset < shader.Spirv.Length;) + { + uint Word(int index) => BinaryPrimitives.ReadUInt32LittleEndian(shader.Spirv.AsSpan(offset + index * 4)); + var header = Word(0); + var opcode = (SpirvOp)(header & 0xFFFF); + if (opcode == SpirvOp.TypeVector && Word(3) == 2) vec2Types.Add(Word(1)); + if (opcode == SpirvOp.TypePointer) pointers.Add(Word(1), (Word(2), Word(3))); + if (opcode == SpirvOp.Variable && vec2Types.Contains(pointers[Word(1)].Pointee)) + packedHalfVariables.Add((Word(1), Word(3))); + offset += (int)(header >> 16) * 4; + } + Assert.NotEmpty(packedHalfVariables); + foreach (var variable in packedHalfVariables) + Assert.Equal(variable.Storage, pointers[variable.Type].Storage); + Gen5LargeDispatcherValidationTests.ValidateWithSpirvToolsWhenAvailable(shader.Spirv); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void PackedFma_WithExactNativeHalfSupport_DoesNotExpandRoundToOdd(bool fmac) + { + var program = fmac + ? Decode([0x78000501u, SEndpgm]) + : Decode([0xCC0E4000u, 257u | (258u << 9) | (259u << 18) | (3u << 27), SEndpgm]); + var (plan, resources, bindings) = ResourceTestProgram.Prepare(program, userDataCount: 0); + var original = new SharpEmu.ShaderCompiler.ShaderCompileRequest(plan, resources, bindings); + var request = new SharpEmu.ShaderCompiler.ShaderCompileRequest(plan, + resources, bindings) { SupportsExactFloat16Conversions = true }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var native, out var error), error); + Assert.True(Gen5SpirvTranslator.TryCompileProgram(original, out var fallback, out error), error); + Gen5LargeDispatcherValidationTests.ValidateWithSpirvToolsWhenAvailable(native.Spirv); + Assert.DoesNotContain((ushort)SpirvOp.FSub, ReadOpcodes(native.Spirv)); + Assert.Contains((ushort)SpirvOp.FSub, ReadOpcodes(fallback.Spirv)); + } + [Fact] public void CompactFloat16ArithmeticDecodesAndCompilesWithoutNativeFloat16() { @@ -138,7 +189,80 @@ public sealed class Gen5Float16ArithmeticTests error); } - private static Gen5ShaderProgram Decode(IReadOnlyList words) + // The native GLSL conversions are used only when the host measured them bit-exact on the + // device; otherwise the branchless integer sequences (FindUMsb = GLSL 75) stay in place. + [Theory] + [InlineData(false)] + [InlineData(true)] + public void PackedFloat16UsesNativeHalfConversionOnlyWhenTheHostMeasuredItExact(bool nativeExact) + { + var program = Decode( + [ + (0x33u << 26) | (0x10u << 16) | 1u, + 0xFFu | (258u << 9), + 0x00002C00u, + SEndpgm, + ]); + + Assert.Equal("VPkMulF16", program.Instructions[0].Opcode); + var (plan, resources, layout) = ResourceTestProgram.Prepare(program, SharpEmu.ShaderCompiler.Resources.ShaderStage.Compute, 0, 0); + var request = new ShaderCompileRequest(plan, resources, layout) + { + NativeHalfConversionExact = nativeExact, + }; + + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + var glsl = ReadGlslExtOpcodes(shader.Spirv); + + // 62 = UnpackHalf2x16, 58 = PackHalf2x16, 75 = FindUMsb. + if (nativeExact) + { + Assert.Contains(62u, glsl); + Assert.Contains(58u, glsl); + } + else + { + Assert.DoesNotContain(62u, glsl); + Assert.DoesNotContain(58u, glsl); + Assert.Contains(75u, glsl); + } + } + + // The GLSL.std.450 instruction number of every ExtInst in the module. + private static IReadOnlyList ReadGlslExtOpcodes(byte[] spirv) + { + var opcodes = new List(); + foreach (var words in ReadInstructionWords(spirv)) + { + if ((ushort)words[0] == (ushort)SpirvOp.ExtInst && words.Length >= 5) + { + opcodes.Add(words[4]); + } + } + + return opcodes; + } + + private static IEnumerable ReadInstructionWords(byte[] spirv) + { + Assert.Equal(0x07230203u, BinaryPrimitives.ReadUInt32LittleEndian(spirv)); + for (var offset = 5 * sizeof(uint); offset < spirv.Length;) + { + var header = BinaryPrimitives.ReadUInt32LittleEndian(spirv.AsSpan(offset)); + var wordCount = checked((int)(header >> 16)); + Assert.InRange(wordCount, 1, (spirv.Length - offset) / sizeof(uint)); + var words = new uint[wordCount]; + for (var index = 0; index < wordCount; index++) + { + words[index] = BinaryPrimitives.ReadUInt32LittleEndian(spirv.AsSpan(offset + index * sizeof(uint))); + } + + yield return words; + offset += wordCount * sizeof(uint); + } + } + + public static Gen5ShaderProgram Decode(IReadOnlyList words) { var memory = new TestCpuMemory(ShaderAddress, words.Count * sizeof(uint)); var bytes = new byte[words.Count * sizeof(uint)]; diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Gen5Int16AluTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Gen5Int16AluTests.cs index 8d2fbbc..dbd0de4 100644 --- a/tests/SharpEmu.ShaderCompiler.Tests/Gen5Int16AluTests.cs +++ b/tests/SharpEmu.ShaderCompiler.Tests/Gen5Int16AluTests.cs @@ -5,6 +5,7 @@ using System.Buffers.Binary; using SharpEmu.HLE; using SharpEmu.ShaderCompiler.Tests.Resources; using SharpEmu.ShaderCompiler.Vulkan; +using SharpEmu.ShaderCompiler.Metal; using Xunit; namespace SharpEmu.ShaderCompiler.Tests; @@ -39,6 +40,23 @@ public sealed class Gen5Int16AluTests Assert.NotEmpty(shader.Spirv); } + [Theory] + [InlineData(0x307u, false)] + [InlineData(0x314u, true)] + public void LogicalHalfWordShiftsPreserveTheOtherDestinationHalfInMetal(uint opcode, bool left) + { + foreach (var select in new uint[] { 0, 1, 2, 3, 8, 11 }) + { + var words = Vop3(opcode); + words[0] |= select << 11; + var program = Decode(words); + Assert.True(Gen5MslTranslator.TryCompileProgram(ResourceTestProgram.Request(program, userDataCount: 0), + out var shader, out var error), error); + Assert.Contains("& 0xFu", shader.Source); + Assert.Contains(left ? "<<" : ">>", shader.Source); + Assert.Contains((select & 8) != 0 ? "v[3] & 0x0000FFFFu" : "v[3] & 0xFFFF0000u", shader.Source); + } + } private static uint[] Vop3(uint opcode) => [(0x35u << 26) | (opcode << 16) | 3u, (0x102u << 9) | 0x101u, SEndpgm]; private static Gen5ShaderProgram Decode(IReadOnlyList words) diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Gen5InterpolationParameterTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Gen5InterpolationParameterTests.cs index ecc4794..f546e44 100644 --- a/tests/SharpEmu.ShaderCompiler.Tests/Gen5InterpolationParameterTests.cs +++ b/tests/SharpEmu.ShaderCompiler.Tests/Gen5InterpolationParameterTests.cs @@ -11,6 +11,103 @@ namespace SharpEmu.ShaderCompiler.Tests; public sealed class Gen5InterpolationParameterTests { + [Theory] + [InlineData(1u)] + [InlineData(16u)] + public void PerSampleCustomInterpolation_UsesSampleIdAndOffsets(uint inputs) + { + var instructions = new Gen5ShaderInstruction[] + { + new(0, Gen5ShaderEncoding.Vintrp, "VInterpMovF32", [0], [Gen5Operand.Vector(0)], + [Gen5Operand.Vector(6)], new Gen5InterpolationControl(1, 0)), + new(4, Gen5ShaderEncoding.Vintrp, "VInterpP2F32", [0], [Gen5Operand.Vector(1)], + [Gen5Operand.Vector(5)], new Gen5InterpolationControl(0, 0)), + ResourceTestProgram.EndProgram(8), + }; + var program = ResourceTestProgram.Program(instructions); + var (plan, resources, layout) = ResourceTestProgram.Prepare(program, ShaderStage.Pixel, userDataCount: 0); + var request = new ShaderCompileRequest(plan, resources, layout) + { + PixelInputAddress = inputs, PixelInputEnable = inputs, PixelInputCntl = [0u, 0u], + PixelRasterizationSamples = 2, + PixelCustomSampleOffsets = [(-.25f, 0f), (.25f, 0f), (-.25f, 0f), (.25f, 0f), + (.25f, 0f), (-.25f, 0f), (.25f, 0f), (-.25f, 0f)], + }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + var spirv = Instructions(shader.Spirv); + Assert.Contains(spirv, instruction => instruction.Opcode == SpirvOp.Decorate && + instruction.Operands[1] == (uint)SpirvDecoration.BuiltIn && + instruction.Operands[2] == (uint)SpirvBuiltIn.SampleId); + Assert.Contains(spirv, instruction => instruction.Opcode == SpirvOp.ExtInst && instruction.Operands[3] == 78); + Assert.DoesNotContain(spirv, instruction => instruction.Opcode == SpirvOp.ExtInst && instruction.Operands[3] == 77); + ValidateWhenAvailable(shader.Spirv); + } + + [Theory] + [InlineData(true, 0xFFFEu, true)] + [InlineData(true, 0xFFFFu, true)] + [InlineData(true, 0u, false)] + [InlineData(false, 0xFFFEu, false)] + public void SampleExclusion_UsesPostDepthCoverageOnlyWithEarlyTests(bool early, uint mask, bool gated) + { + var program = ResourceTestProgram.Program(ResourceTestProgram.EndProgram(0)); + var (plan, resources, layout) = ResourceTestProgram.Prepare(program, ShaderStage.Pixel, userDataCount: 0); + var request = new ShaderCompileRequest(plan, resources, layout) + { + EarlyFragmentTests = early, + PixelShaderSampleExclusionMask = mask, + PixelRasterizationSamples = 2, + }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + var instructions = Instructions(shader.Spirv); + Assert.Equal(gated, instructions.Any(instruction => instruction.Opcode == SpirvOp.ExecutionMode && + instruction.Operands[1] == (uint)SpirvExecutionMode.PostDepthCoverage)); + Assert.Equal(gated, instructions.Any(instruction => instruction.Opcode == SpirvOp.Decorate && + instruction.Operands[1] == (uint)SpirvDecoration.BuiltIn && + instruction.Operands[2] == (uint)SpirvBuiltIn.SampleMask)); + ValidateWhenAvailable(shader.Spirv); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void FixedCustomSampleInterpolation_UsesOffsetsInsteadOfSampleLookup(bool linear) + { + var instruction = new Gen5ShaderInstruction(0, Gen5ShaderEncoding.Vintrp, "VInterpP2F32", + [0], [Gen5Operand.Vector(1)], [Gen5Operand.Vector(5)], new Gen5InterpolationControl(0, 0)); + var program = ResourceTestProgram.Program(instruction, ResourceTestProgram.EndProgram(4)); + var (plan, resources, layout) = ResourceTestProgram.Prepare(program, ShaderStage.Pixel, userDataCount: 0); + var request = new ShaderCompileRequest(plan, resources, layout) + { + PixelInputAddress = linear ? 16u : 1u, PixelInputEnable = linear ? 16u : 1u, + PixelInputCntl = [0u], PixelInterpolationSample = 0, + PixelCustomSampleOffsets = [(-.25f, 0f), (-.25f, 0f), (.25f, 0f), (.25f, 0f)], + }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + var instructions = Instructions(shader.Spirv); + Assert.Contains(instructions, instruction => instruction.Opcode == SpirvOp.ExtInst && instruction.Operands[3] == 78); + Assert.DoesNotContain(instructions, instruction => instruction.Opcode == SpirvOp.ExtInst && instruction.Operands[3] == 77); + ValidateWhenAvailable(shader.Spirv); + } + + [Theory] + [InlineData(3, 0f, true)] + [InlineData(4, float.NaN, true)] + [InlineData(4, .75f, true)] + [InlineData(4, 0f, false)] + public void FixedCustomSampleInterpolation_RejectsUnsupportedGrid(int count, float x, bool fixedSample) + { + var program = ResourceTestProgram.Program(ResourceTestProgram.EndProgram(0)); + var (plan, resources, layout) = ResourceTestProgram.Prepare(program, ShaderStage.Pixel, userDataCount: 0); + var request = new ShaderCompileRequest(plan, resources, layout) + { + PixelInterpolationSample = fixedSample ? 0u : null, + PixelCustomSampleOffsets = Enumerable.Repeat((x, 0f), count).ToArray(), + }; + Assert.False(Gen5SpirvTranslator.TryCompileProgram(request, out _, out var error)); + Assert.Contains("four finite offsets", error); + } + [Theory] [InlineData(0u, false, 1u, true)] [InlineData(1u, false, 2u, true)] @@ -72,6 +169,68 @@ public sealed class Gen5InterpolationParameterTests ValidateWhenAvailable(shader.Spirv); } + [Theory] + [InlineData(1u, false)] + [InlineData(0x10u, true)] + public void SingleSampleModeQualifiesOrdinaryInterpolants(uint inputs, bool noPerspective) + { + Assert.True(Gen5SpirvTranslator.TryCompileProgram( + Request(0, false, inputs, "VInterpP2F32", inputCntl: 0), out var shader, out var error), error); + var instructions = Instructions(shader.Spirv); + var sample = Assert.Single(instructions, item => item.Opcode == SpirvOp.Decorate && + item.Operands[1] == (uint)SpirvDecoration.Sample); + Assert.Equal(noPerspective, instructions.Any(item => item.Opcode == SpirvOp.Decorate && + item.Operands[0] == sample.Operands[0] && item.Operands[1] == (uint)SpirvDecoration.NoPerspective)); + ValidateWhenAvailable(shader.Spirv); + } + + [Theory] + [InlineData(1u, "VInterpP2F32", 0u)] + [InlineData(0x10u, "VInterpP2F32", 1u)] + [InlineData(1u, "VInterpMovF32", 0u)] + [InlineData(0x77u, "VInterpMovF32", 1u)] + public void FixedSampleInterpolation_DoesNotRequestSampleInvocations(uint inputs, string opcode, uint sample) + { + var request = Request(2, true, inputs, opcode, inputCntl: 0, fixedSample: sample); + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + var instructions = Instructions(shader.Spirv); + Assert.DoesNotContain(instructions, item => item.Opcode == SpirvOp.Decorate && + (item.Operands[1] == (uint)SpirvDecoration.Sample || + (item.Operands[1] == (uint)SpirvDecoration.BuiltIn && item.Operands[2] == (uint)SpirvBuiltIn.SampleId))); + var interpolations = instructions.Where(item => item.Opcode == SpirvOp.ExtInst && item.Operands[3] == 77).ToArray(); + Assert.NotEmpty(interpolations); + foreach (var interpolation in interpolations) + Assert.Contains(instructions, item => item.Opcode == SpirvOp.Constant && + item.Operands[1] == interpolation.Operands[5] && item.Operands[2] == sample); + ValidateWhenAvailable(shader.Spirv); + } + + [Theory] + [InlineData(3u, 0u)] + [InlineData(0x11u, 0u)] + [InlineData(1u, 0x400u)] + [InlineData(0x10u, 0x400u)] + public void MixedModesAndFlatInputsAreNotSampleQualified(uint inputs, uint inputCntl) + { + Assert.True(Gen5SpirvTranslator.TryCompileProgram( + Request(0, false, inputs, "VInterpP2F32", inputCntl), out var shader, out var error), error); + Assert.DoesNotContain(Instructions(shader.Spirv), item => item.Opcode == SpirvOp.Decorate && + item.Operands[1] == (uint)SpirvDecoration.Sample); + ValidateWhenAvailable(shader.Spirv); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void EarlyDepthRequest_ControlsFragmentExecutionMode(bool earlyTests) + { + var request = Request(0, false, opcode: "VInterpP2F32", earlyTests: earlyTests); + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + Assert.Equal(earlyTests, Instructions(shader.Spirv).Any(item => item.Opcode == SpirvOp.ExecutionMode && + item.Operands[1] == (uint)SpirvExecutionMode.EarlyFragmentTests)); + ValidateWhenAvailable(shader.Spirv); + } + [Fact] public void PullModel_DoesNotSilentlyUseZeroCoordinates() { @@ -160,8 +319,10 @@ public sealed class Gen5InterpolationParameterTests ValidateWhenAvailable(shader.Spirv); } - [Fact] - public void SmoothSlotSharingAPerVertexParameter_InterpolatesThePerVertexInput() + [Theory] + [InlineData(null)] + [InlineData(0u)] + public void SmoothSlotSharingAPerVertexParameter_InterpolatesThePerVertexInput(uint? fixedSample) { // Slot 1 is read per vertex, slot 2 interpolates the same VS parameter: one per-vertex // input serves both, and slot 2 is rebuilt from the vertices with the barycentrics. @@ -173,8 +334,9 @@ public sealed class Gen5InterpolationParameterTests var (plan, resources, layout) = ResourceTestProgram.Prepare(program, ShaderStage.Pixel, userDataCount: 0); var request = new ShaderCompileRequest(plan, resources, layout) { - PixelInputAddress = 2, - PixelInputEnable = 2, + PixelInputAddress = fixedSample.HasValue ? 1u : 2u, + PixelInputEnable = fixedSample.HasValue ? 1u : 2u, + PixelInterpolationSample = fixedSample, PixelInputCntl = [0, 0x1, 0x1], PixelCustomInterpolationMask = 2, SupportsPerVertexPixelInputs = true, @@ -190,6 +352,29 @@ public sealed class Gen5InterpolationParameterTests instruction.Operands[1] == (uint)SpirvDecoration.BuiltIn && instruction.Operands[2] == (uint)SpirvBuiltIn.BaryCoordKhr); Assert.Equal(4, instructions.Count(instruction => instruction.Opcode == SpirvOp.AccessChain && instruction.Operands[2] == input)); + if (fixedSample.HasValue) + { + Assert.Contains(instructions, item => item.Opcode == SpirvOp.ExtInst && item.Operands[3] == 77); + Assert.DoesNotContain(instructions, item => item.Opcode == SpirvOp.Decorate && + item.Operands[1] == (uint)SpirvDecoration.BuiltIn && item.Operands[2] == (uint)SpirvBuiltIn.SampleId); + } + ValidateWhenAvailable(shader.Spirv); + } + + [Fact] + public void FrontFace_ReadsTheRasterizerBuiltIn() + { + Assert.True(Gen5SpirvTranslator.TryCompileProgram( + Request(0, false, inputs: 0x1002, opcode: "VInterpP2F32"), out var shader, out var error), error); + var instructions = Instructions(shader.Spirv); + var frontFace = Assert.Single(instructions, instruction => + instruction.Opcode == SpirvOp.Decorate && + instruction.Operands[1] == (uint)SpirvDecoration.BuiltIn && + instruction.Operands[2] == (uint)SpirvBuiltIn.FrontFacing).Operands[0]; + Assert.Contains(instructions, instruction => instruction.Opcode == SpirvOp.Load && + instruction.Operands[2] == frontFace); + Assert.Contains(instructions, instruction => instruction.Opcode == SpirvOp.Constant && + instruction.Operands.Length == 3 && instruction.Operands[2] == 0x3F800000u); ValidateWhenAvailable(shader.Spirv); } @@ -254,9 +439,108 @@ public sealed class Gen5InterpolationParameterTests ValidateWhenAvailable(shader.Spirv); } + [Theory] + [InlineData(false, 1u, 1u)] + [InlineData(false, 2u, 2u)] + [InlineData(false, 1u, 2u)] + [InlineData(true, 2u, 2u)] + [InlineData(true, 1u, 2u)] + public void MrtzSampleMask_DeclaresAnIntegerOutput(bool compressed, uint exportSamples, uint rasterSamples) + { + var request = MaskExportRequest(compressed, exportSamples, rasterSamples); + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + var instructions = Instructions(shader.Spirv); + var output = Assert.Single(instructions, instruction => instruction.Opcode == SpirvOp.Decorate && + instruction.Operands[1] == (uint)SpirvDecoration.BuiltIn && + instruction.Operands[2] == (uint)SpirvBuiltIn.SampleMask).Operands[0]; + Assert.Contains(instructions, instruction => instruction.Opcode == SpirvOp.Variable && + instruction.Operands[1] == output && instruction.Operands[2] == (uint)SpirvStorageClass.Output); + var pointers = instructions.Where(instruction => instruction.Opcode == SpirvOp.AccessChain && + instruction.Operands[2] == output).Select(instruction => instruction.Operands[1]).ToHashSet(); + Assert.Equal(2, instructions.Count(instruction => instruction.Opcode == SpirvOp.Store && + pointers.Contains(instruction.Operands[0]))); + Assert.Contains(instructions, instruction => instruction.Opcode == SpirvOp.Select); + ValidateWhenAvailable(shader.Spirv); + } + + [Theory] + [InlineData(0u, 2u)] + [InlineData(2u, 3u)] + [InlineData(2u, 4u)] + [InlineData(4u, 2u)] + public void MrtzSampleMask_RejectsUnverifiedCountMappings(uint exportSamples, uint rasterSamples) + { + Assert.False(Gen5SpirvTranslator.TryCompileProgram( + MaskExportRequest(false, exportSamples, rasterSamples), out _, out var error)); + Assert.Contains("unsupported pixel sample-mask export counts", error); + } + + private static ShaderCompileRequest MaskExportRequest(bool compressed, uint exportSamples, uint rasterSamples) + { + var export = new Gen5ShaderInstruction(0, Gen5ShaderEncoding.Exp, "Exp", [], + [Gen5Operand.Vector(0), Gen5Operand.Vector(1), Gen5Operand.Vector(2), Gen5Operand.Vector(3)], [], + new Gen5ExportControl(8, compressed ? 12u : 4u, compressed, true, true)); + var program = ResourceTestProgram.Program(export, ResourceTestProgram.EndProgram(8)); + var (plan, resources, layout) = ResourceTestProgram.Prepare(program, ShaderStage.Pixel, userDataCount: 0); + return new ShaderCompileRequest(plan, resources, layout) + { + PixelSampleMaskExportEnable = true, + PixelMaskExportSamples = exportSamples, + PixelRasterizationSamples = rasterSamples, + }; + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void MrtzDepth_PreservesDepthAndAnOptionalSampleMask(bool sampleMask) + { + Assert.True(Gen5SpirvTranslator.TryCompileProgram( + DepthExportRequest(sampleMask), out var shader, out var error), error); + var instructions = Instructions(shader.Spirv); + var depth = Assert.Single(instructions, instruction => instruction.Opcode == SpirvOp.Decorate && + instruction.Operands[1] == (uint)SpirvDecoration.BuiltIn && + instruction.Operands[2] == (uint)SpirvBuiltIn.FragDepth).Operands[0]; + Assert.Equal(2, instructions.Count(instruction => instruction.Opcode == SpirvOp.Store && + instruction.Operands[0] == depth)); + Assert.Contains(instructions, instruction => instruction.Opcode == SpirvOp.ExecutionMode && + instruction.Operands[1] == (uint)SpirvExecutionMode.DepthReplacing); + Assert.DoesNotContain(instructions, instruction => instruction.Opcode == SpirvOp.ExecutionMode && + instruction.Operands[1] == (uint)SpirvExecutionMode.EarlyFragmentTests); + Assert.Equal(sampleMask, instructions.Any(instruction => instruction.Opcode == SpirvOp.Decorate && + instruction.Operands[1] == (uint)SpirvDecoration.BuiltIn && + instruction.Operands[2] == (uint)SpirvBuiltIn.SampleMask)); + ValidateWhenAvailable(shader.Spirv); + } + + [Theory] + [InlineData(true, false, "late fragment tests")] + [InlineData(false, true, "compressed MRTZ")] + public void MrtzDepth_RejectsIncompatibleState(bool early, bool compressed, string reason) + { + Assert.False(Gen5SpirvTranslator.TryCompileProgram( + DepthExportRequest(false, early, compressed), out _, out var error)); + Assert.Contains(reason, error); + } + + private static ShaderCompileRequest DepthExportRequest(bool sampleMask, bool early = false, bool compressed = false) + { + var export = new Gen5ShaderInstruction(0, Gen5ShaderEncoding.Exp, "Exp", [], + [Gen5Operand.Vector(0), Gen5Operand.Vector(1), Gen5Operand.Vector(2), Gen5Operand.Vector(3)], [], + new Gen5ExportControl(8, sampleMask ? 5u : 1u, compressed, true, true)); + var program = ResourceTestProgram.Program(export, ResourceTestProgram.EndProgram(8)); + var (plan, resources, layout) = ResourceTestProgram.Prepare(program, ShaderStage.Pixel, userDataCount: 0); + return new ShaderCompileRequest(plan, resources, layout) + { + PixelDepthExportEnable = true, + PixelSampleMaskExportEnable = sampleMask, + EarlyFragmentTests = early, + }; + } + private static ShaderCompileRequest Request( uint selector, bool custom, uint inputs = 2, string opcode = "VInterpMovF32", - uint inputCntl = 0x401, bool supportsPerVertex = true) + uint inputCntl = 0x401, bool supportsPerVertex = true, uint? fixedSample = null, bool earlyTests = false) { var interpolation = new Gen5ShaderInstruction(0, Gen5ShaderEncoding.Vintrp, opcode, [selector], [Gen5Operand.Vector(selector)], [Gen5Operand.Vector(4)], new Gen5InterpolationControl(1, 2)); @@ -266,6 +550,8 @@ public sealed class Gen5InterpolationParameterTests { PixelInputAddress = inputs, PixelInputEnable = inputs, + PixelInterpolationSample = fixedSample, + EarlyFragmentTests = earlyTests, PixelInputCntl = [0, inputCntl], PixelCustomInterpolationMask = custom ? 2u : 0u, SupportsPerVertexPixelInputs = supportsPerVertex, @@ -289,12 +575,14 @@ public sealed class Gen5InterpolationParameterTests private static void ValidateWhenAvailable(byte[] code) { var sdk = Environment.GetEnvironmentVariable("VULKAN_SDK"); - if (string.IsNullOrWhiteSpace(sdk)) - { - return; - } - var executable = Path.Combine(sdk, OperatingSystem.IsWindows() ? "Bin/spirv-val.exe" : "bin/spirv-val"); - if (!File.Exists(executable)) + var name = OperatingSystem.IsWindows() ? "spirv-val.exe" : "spirv-val"; + var sdkExecutable = string.IsNullOrWhiteSpace(sdk) ? null : + Path.Combine(sdk, OperatingSystem.IsWindows() ? "Bin" : "bin", name); + var executable = sdkExecutable is not null && File.Exists(sdkExecutable) ? sdkExecutable : + (Environment.GetEnvironmentVariable("PATH") ?? string.Empty) + .Split(Path.PathSeparator, StringSplitOptions.RemoveEmptyEntries) + .Select(directory => Path.Combine(directory.Trim('"'), name)).FirstOrDefault(File.Exists); + if (executable is null) { return; } diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Gen5LargeDispatcherValidationTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Gen5LargeDispatcherValidationTests.cs new file mode 100644 index 0000000..ff6cca8 --- /dev/null +++ b/tests/SharpEmu.ShaderCompiler.Tests/Gen5LargeDispatcherValidationTests.cs @@ -0,0 +1,252 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Diagnostics; +using SharpEmu.ShaderCompiler.Vulkan; +using SharpEmu.ShaderCompiler.Tests.Resources; +using Xunit; + +namespace SharpEmu.ShaderCompiler.Tests; + +public sealed class Gen5LargeDispatcherValidationTests +{ + [Fact] + public void LargeDispatcherProducesValidStructuredSpirv() + { + const int branchPairs = 400; + var instructions = new List(branchPairs * 2 + 1); + for (var index = 0; index < branchPairs; index++) + { + var pc = (uint)index * 8; + instructions.Add(ResourceTestProgram.Branch(pc, "SCbranchScc0", 1)); + instructions.Add(ResourceTestProgram.MoveScalar(pc + 4, 0, (uint)index)); + } + + instructions.Add(ResourceTestProgram.EndProgram((uint)branchPairs * 8)); + var request = ResourceTestProgram.Request( + new Gen5ShaderProgram(0, instructions), + userDataCount: 0); + + Assert.True( + Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), + error); + Assert.Contains( + ReadSpirvOpcodes(shader.Spirv), + opcode => opcode == SpirvOp.SelectionMerge); + + ValidateStructuredControlFlow(shader.Spirv); + ValidateWithSpirvToolsWhenAvailable(shader.Spirv); + } + + [Fact] + public void ScalarBitCount64Uses32BitSpirvOperands() + { + var request = ResourceTestProgram.Request( + ResourceTestProgram.Program( + ResourceTestProgram.MoveScalar(0, 0, 0x0123_4567), + ResourceTestProgram.MoveScalar(4, 1, 0x89AB_CDEF), + ResourceTestProgram.Sop1( + 8, + "SBcnt1I32B64", + 2, + Gen5Operand.Scalar(0)), + ResourceTestProgram.EndProgram(12)), + userDataCount: 0); + + Assert.True( + Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), + error); + + ValidateBitCountOperandTypes(shader.Spirv); + ValidateWithSpirvToolsWhenAvailable(shader.Spirv); + } + + [Fact] + public void DataShareWaveCounterUses32BitBitCountOperands() + { + var request = ResourceTestProgram.Request( + ResourceTestProgram.Program( + ResourceTestProgram.MoveScalar(0, 124, 0x0001_0004), + ResourceTestProgram.DataShare( + 4, + "DsAppend", + gds: false, + [Gen5Operand.Scalar(124)], + [1]), + ResourceTestProgram.EndProgram(8)), + userDataCount: 0); + + Assert.True( + Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), + error); + + ValidateBitCountOperandTypes(shader.Spirv); + ValidateWithSpirvToolsWhenAvailable(shader.Spirv); + } + + internal static IReadOnlyList ReadSpirvOpcodes(byte[] code) + { + var words = new uint[code.Length / sizeof(uint)]; + Buffer.BlockCopy(code, 0, words, 0, code.Length); + var result = new List(); + for (var index = 5; index < words.Length;) + { + var instructionWord = words[index]; + var wordCount = (int)(instructionWord >> 16); + Assert.True(wordCount > 0 && index + wordCount <= words.Length); + result.Add((SpirvOp)(instructionWord & 0xFFFF)); + index += wordCount; + } + + return result; + } + + private static IReadOnlyList ReadSpirvInstructions(byte[] code) + { + var words = new uint[code.Length / sizeof(uint)]; + Buffer.BlockCopy(code, 0, words, 0, code.Length); + var result = new List(); + for (var index = 5; index < words.Length;) + { + var instructionWord = words[index]; + var wordCount = (int)(instructionWord >> 16); + Assert.True(wordCount > 0 && index + wordCount <= words.Length); + result.Add(new ParsedInstruction( + (SpirvOp)(instructionWord & 0xFFFF), + words[(index + 1)..(index + wordCount)])); + index += wordCount; + } + + return result; + } + + internal static void ValidateStructuredControlFlow(byte[] code) + { + var blockOpen = false; + var blockTerminated = false; + var mergeCount = 0; + foreach (var instruction in ReadSpirvInstructions(code)) + { + if (instruction.Opcode == SpirvOp.FunctionEnd) + { + if (blockOpen) + { + Assert.True(blockTerminated, "A function ended with an unterminated basic block."); + blockOpen = false; + } + + continue; + } + + if (instruction.Opcode == SpirvOp.Label) + { + if (blockOpen) + { + Assert.True(blockTerminated, "An OpLabel followed an unterminated basic block."); + } + + blockOpen = true; + blockTerminated = false; + mergeCount = 0; + continue; + } + + if (!blockOpen) + { + continue; + } + + if (instruction.Opcode is SpirvOp.SelectionMerge or SpirvOp.LoopMerge) + { + Assert.True(++mergeCount == 1, "A basic block contains more than one merge instruction."); + } + + if (IsTerminator(instruction.Opcode)) + { + Assert.False(blockTerminated, "A basic block contains instructions after its terminator."); + blockTerminated = true; + } + else + { + Assert.False(blockTerminated, "A basic block contains an instruction after its terminator."); + } + } + + Assert.False(blockOpen, "The SPIR-V module ended inside a basic block."); + } + + private static void ValidateBitCountOperandTypes(byte[] code) + { + var instructions = ReadSpirvInstructions(code); + var integerWidths = instructions + .Where(instruction => instruction.Opcode == SpirvOp.TypeInt) + .ToDictionary(instruction => instruction.Operands[0], instruction => instruction.Operands[1]); + var resultTypes = instructions + .Where(instruction => HasResultType(instruction.Opcode) && instruction.Operands.Length >= 2) + .ToDictionary(instruction => instruction.Operands[1], instruction => instruction.Operands[0]); + var bitCounts = instructions.Where(instruction => instruction.Opcode == SpirvOp.BitCount).ToArray(); + + Assert.NotEmpty(bitCounts); + foreach (var bitCount in bitCounts) + { + Assert.True(integerWidths.TryGetValue(bitCount.Operands[0], out var resultWidth)); + Assert.Equal(32u, resultWidth); + Assert.True(resultTypes.TryGetValue(bitCount.Operands[2], out var operandType)); + Assert.True(integerWidths.TryGetValue(operandType, out var operandWidth)); + Assert.Equal(32u, operandWidth); + } + } + + private static bool HasResultType(SpirvOp opcode) => opcode is + SpirvOp.BitCount or SpirvOp.Bitcast or SpirvOp.IAdd or SpirvOp.ISub or + SpirvOp.ShiftRightLogical or SpirvOp.UConvert or SpirvOp.Constant; + + private static bool IsTerminator(SpirvOp opcode) => opcode is + SpirvOp.Branch or SpirvOp.BranchConditional or SpirvOp.Switch or SpirvOp.Kill or + SpirvOp.Return or SpirvOp.ReturnValue or SpirvOp.Unreachable || + (ushort)opcode is 4416 or 4448 or 4449; // OpTerminateInvocation/IgnoreIntersectionKHR/TerminateRayKHR. + + private sealed record ParsedInstruction(SpirvOp Opcode, uint[] Operands); + + internal static void ValidateWithSpirvToolsWhenAvailable(byte[] code) + { + var sdk = Environment.GetEnvironmentVariable("VULKAN_SDK"); + var candidates = new List(); + if (!string.IsNullOrWhiteSpace(sdk)) + { + candidates.Add(Path.Combine(sdk, OperatingSystem.IsWindows() ? "Bin/spirv-val.exe" : "bin/spirv-val")); + } + + candidates.Add(@"C:\VulkanSDK\1.4.357.0\Bin\spirv-val.exe"); + var executable = candidates.FirstOrDefault(File.Exists); + if (executable is null) + { + return; + } + + var path = Path.GetTempFileName(); + try + { + File.WriteAllBytes(path, code); + var start = new ProcessStartInfo(executable) + { + UseShellExecute = false, + RedirectStandardError = true, + RedirectStandardOutput = true, + CreateNoWindow = true, + }; + start.ArgumentList.Add("--target-env"); + start.ArgumentList.Add("vulkan1.2"); + start.ArgumentList.Add(path); + using var process = Process.Start(start)!; + var output = process.StandardOutput.ReadToEnd(); + var error = process.StandardError.ReadToEnd(); + process.WaitForExit(); + Assert.True(process.ExitCode == 0, $"spirv-val failed: {output}{error}"); + } + finally + { + File.Delete(path); + } + } +} diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Gen5PixelOutputMappingTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Gen5PixelOutputMappingTests.cs index b28e90e..bef92f6 100644 --- a/tests/SharpEmu.ShaderCompiler.Tests/Gen5PixelOutputMappingTests.cs +++ b/tests/SharpEmu.ShaderCompiler.Tests/Gen5PixelOutputMappingTests.cs @@ -52,6 +52,26 @@ public sealed class Gen5PixelOutputMappingTests instruction => instruction.Opcode == SpirvOp.VectorShuffle); } + [Theory] + [InlineData(Gen5PixelExportFormat.Unorm16, "/ 65535.0f")] + [InlineData(Gen5PixelExportFormat.Snorm16, "/ 32767.0f")] + [InlineData(Gen5PixelExportFormat.Uint16, "& 0xffffu")] + [InlineData(Gen5PixelExportFormat.Sint16, "as_type")] + public void MetalCompressedExportUsesTheDeclaredRepresentation(Gen5PixelExportFormat format, string expected) + { + var program = ResourceTestProgram.Program(new Gen5ShaderInstruction(0, Gen5ShaderEncoding.Exp, + "Exp", [], [Gen5Operand.Vector(0), Gen5Operand.Vector(1), Gen5Operand.Vector(0), Gen5Operand.Vector(0)], + [], new Gen5ExportControl(0, 15, true, true, true)), ResourceTestProgram.EndProgram(8)); + var (plan, resources, layout) = ResourceTestProgram.Prepare(program, ShaderStage.Pixel, userDataCount: 0); + var request = new ShaderCompileRequest(plan, resources, layout) + { + PixelOutputs = [new Gen5PixelOutputBinding(0, 0, Gen5PixelOutputKind.Float) { ExportFormat = format }], + }; + Assert.True(Gen5MslTranslator.TryCompileProgram(request, out var shader, out var error), error); + Assert.Contains(expected, shader.Source); + Assert.DoesNotContain("as_type", shader.Source); + } + [Fact] public void BgraOutputShufflesGuestComponentsToPhysicalOrder() { diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Gen5PixelReadlaneTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Gen5PixelReadlaneTests.cs new file mode 100644 index 0000000..1bd5523 --- /dev/null +++ b/tests/SharpEmu.ShaderCompiler.Tests/Gen5PixelReadlaneTests.cs @@ -0,0 +1,54 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Buffers.Binary; +using SharpEmu.ShaderCompiler; +using SharpEmu.ShaderCompiler.Resources; +using SharpEmu.ShaderCompiler.Tests.Resources; +using SharpEmu.ShaderCompiler.Vulkan; +using Xunit; + +namespace SharpEmu.ShaderCompiler.Tests; + +// A graphics wave is one host subgroup of up to 32 guest lanes. V_READLANE in a pixel shader +// must read the selected lane of that group: Ghost of Yotei reduces a wave minimum with it, +// and a loop that waits for the minimum to reach 255 never ends on the reading lane's own value. +public sealed class Gen5PixelReadlaneTests +{ + private static Gen5ShaderInstruction Export(uint pc) => + new(pc, Gen5ShaderEncoding.Exp, "Exp", [0u, 0u], + [Gen5Operand.Vector(4), Gen5Operand.Vector(4), Gen5Operand.Vector(4), Gen5Operand.Vector(4)], [], + new Gen5ExportControl(0, 15, false, true, false)); + + private static IEnumerable Opcodes(byte[] bytes) + { + for (var offset = 20; offset < bytes.Length;) + { + var header = BinaryPrimitives.ReadUInt32LittleEndian(bytes.AsSpan(offset)); + yield return header & 0xFFFF; + offset += (int)(header >> 16) * 4; + } + } + + [Theory] + [InlineData(31u)] + [InlineData(63u)] + public void Readlane_ReadsTheSelectedLaneOfTheSubgroup(uint lane) + { + var program = ResourceTestProgram.Program( + ResourceTestProgram.ReadLane(0, 20, 3, lane), + ResourceTestProgram.MoveVectorFromScalar(8, 4, 20), + Export(12), + ResourceTestProgram.EndProgram(20)); + var (plan, resources, layout) = ResourceTestProgram.Prepare(program, ShaderStage.Pixel, userDataCount: 0); + var request = new ShaderCompileRequest(plan, resources, layout) + { + PixelOutputs = [new Gen5PixelOutputBinding(0, 0, Gen5PixelOutputKind.Float)], + }; + + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + Gen5LargeDispatcherValidationTests.ValidateWithSpirvToolsWhenAvailable(shader.Spirv); + const uint GroupNonUniformShuffle = 345; + Assert.Contains(GroupNonUniformShuffle, Opcodes(shader.Spirv)); + } +} diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Gen5PrivateLdsTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Gen5PrivateLdsTests.cs new file mode 100644 index 0000000..94906c6 --- /dev/null +++ b/tests/SharpEmu.ShaderCompiler.Tests/Gen5PrivateLdsTests.cs @@ -0,0 +1,105 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.ShaderCompiler.Resources; +using SharpEmu.ShaderCompiler.Tests.Resources; +using SharpEmu.ShaderCompiler.Vulkan; +using Xunit; +using static SharpEmu.ShaderCompiler.Tests.Resources.ResourceTestProgram; + +namespace SharpEmu.ShaderCompiler.Tests; + +// Graphics stages model LDS as a per-invocation private array that reads zero where the +// invocation has not written. A written-dword bitmap provides that instead of a zero +// initializer over the whole array. +public sealed class Gen5PrivateLdsTests +{ + private const uint PrivateStorageClass = 6; + private const uint WorkgroupStorageClass = 4; + + [Fact] + public void VertexStageLdsIsLazilyZeroedThroughAWrittenBitmap() + { + var spirv = Compile(ShaderStage.Vertex); + + var module = new SpirvModuleInspector(spirv); + var lds = module.Names.Single(pair => pair.Value == "lds").Key; + var written = module.Names.Single(pair => pair.Value == "ldsWritten").Key; + Assert.Equal(PrivateStorageClass, module.VariableStorageClasses[lds]); + Assert.Equal(PrivateStorageClass, module.VariableStorageClasses[written]); + Assert.False(HasInitializer(spirv, lds)); + Assert.True(HasInitializer(spirv, written)); + ValidateWhenAvailable(spirv); + } + + [Fact] + public void ComputeLdsStaysWorkgroupMemoryWithoutABitmap() + { + var spirv = Compile(ShaderStage.Compute); + + var module = new SpirvModuleInspector(spirv); + var lds = module.Names.Single(pair => pair.Value == "lds").Key; + Assert.Equal(WorkgroupStorageClass, module.VariableStorageClasses[lds]); + Assert.DoesNotContain("ldsWritten", module.Names.Values); + } + + private static byte[] Compile(ShaderStage stage) + { + var program = Program( + Vop2(0, "VLshlrevB32", 3, Operand(2), Gen5Operand.Vector(0)), + DataShare(4, "DsWriteB32", false, [Gen5Operand.Vector(3), Gen5Operand.Vector(0)], []), + DataShare(12, "DsReadB32", false, [Gen5Operand.Vector(3)], [5], 4), + EndProgram(20)); + var request = Request(program, stage); + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + return shader.Spirv; + } + + private static void ValidateWhenAvailable(byte[] code) + { + var sdk = Environment.GetEnvironmentVariable("VULKAN_SDK"); + var executable = string.IsNullOrWhiteSpace(sdk) ? null + : Path.Combine(sdk, OperatingSystem.IsWindows() ? "Bin/spirv-val.exe" : "bin/spirv-val"); + if (executable is null || !File.Exists(executable)) + { + return; + } + + var path = Path.GetTempFileName(); + try + { + File.WriteAllBytes(path, code); + var start = new System.Diagnostics.ProcessStartInfo(executable) { UseShellExecute = false, RedirectStandardError = true, CreateNoWindow = true }; + start.ArgumentList.Add("--target-env"); + start.ArgumentList.Add("vulkan1.2"); + start.ArgumentList.Add(path); + using var process = System.Diagnostics.Process.Start(start)!; + var error = process.StandardError.ReadToEnd(); + process.WaitForExit(); + Assert.True(process.ExitCode == 0, error); + } + finally + { + File.Delete(path); + } + } + + // OpVariable carries an initializer operand after its storage class. + private static bool HasInitializer(byte[] spirv, uint variable) + { + var words = new uint[spirv.Length / 4]; + Buffer.BlockCopy(spirv, 0, words, 0, spirv.Length); + for (var offset = 5; offset < words.Length;) + { + var wordCount = (int)(words[offset] >> 16); + if ((SpirvOp)(words[offset] & 0xFFFF) == SpirvOp.Variable && words[offset + 2] == variable) + { + return wordCount > 4; + } + + offset += Math.Max(wordCount, 1); + } + + throw new InvalidOperationException($"Variable {variable} is not declared."); + } +} diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Gen5RenderScaleTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Gen5RenderScaleTests.cs new file mode 100644 index 0000000..75c9156 --- /dev/null +++ b/tests/SharpEmu.ShaderCompiler.Tests/Gen5RenderScaleTests.cs @@ -0,0 +1,255 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Buffers.Binary; +using System.Diagnostics; +using SharpEmu.ShaderCompiler.Resources; +using SharpEmu.ShaderCompiler.Tests.Resources; +using SharpEmu.ShaderCompiler.Vulkan; +using Xunit; + +namespace SharpEmu.ShaderCompiler.Tests; + +// Internal resolution scaling: the module has to translate the pixel position, integer +// image coordinates and size queries between guest pixels and host texels. +public sealed class Gen5RenderScaleTests +{ + private const uint PositionXBit = 1u << 8; + private const uint PositionYBit = 1u << 9; + private const uint PositionZBit = 1u << 10; + + [Fact] + public void RenderScaleDwordsFollowTheDispatchLimits() + { + var info = new ShaderResourceInfo(); + var plain = BindingLayout.Allocate(info, [0u], false, false, false); + var scaled = BindingLayout.Allocate(info, [0u], false, false, false, 0, false, usesRenderScale: true); + var both = BindingLayout.Allocate(info, [0u], false, false, false, 0, true, usesRenderScale: true); + + Assert.False(plain.UsesRenderScale); + Assert.True(scaled.UsesRenderScale); + Assert.Equal(plain.ShaderDataDwordCount + BindingLayout.RenderScaleDwordCount, scaled.ShaderDataDwordCount); + Assert.Equal(plain.ShaderDataDwordCount, scaled.RenderScaleDword); + Assert.Equal(plain.ShaderDataDwordCount + 3, both.RenderScaleDword); + Assert.NotEqual(plain, scaled); + + // The validator recomputes the layout from the flag, so declarations never drift. + BindingLayoutValidator.Validate(plain, info, [0u], false, false, false, 0, ShaderStage.Pixel); + BindingLayoutValidator.Validate(scaled, info, [0u], false, false, false, 0, ShaderStage.Pixel); + } + + [Theory] + [InlineData(true)] + [InlineData(false)] + public void PixelPositionIsDividedByTheAttachmentFactor(bool usesRenderScale) + { + var spirv = CompilePixelPosition(usesRenderScale); + var instructions = Parse(spirv); + var fragCoord = Load(instructions, FragCoordVariable(instructions)); + + // Only the two scaled axes move; depth and 1/w keep their values. + Assert.Equal(usesRenderScale, FeedsOperation(instructions, fragCoord, 0, SpirvOp.FMul)); + Assert.Equal(usesRenderScale, FeedsOperation(instructions, fragCoord, 1, SpirvOp.FMul)); + Assert.False(FeedsOperation(instructions, fragCoord, 2, SpirvOp.FMul)); + ValidateWhenAvailable(spirv); + } + + [Theory] + [InlineData(true)] + [InlineData(false)] + public void IntegerImageCoordinatesAreMappedOntoHostTexels(bool usesRenderScale) + { + var spirv = CompileImageAccess("ImageLoad", usesRenderScale); + var instructions = Parse(spirv); + var fetch = Assert.Single(instructions, item => item.Opcode == SpirvOp.ImageFetch); + var coordinates = Definition(instructions, fetch.Operands[3]); + + // Two coordinates, each either the raw register or that register mapped through + // the image's factor; a selected factor of one leaves an unscaled image alone. + Assert.Equal(SpirvOp.CompositeConstruct, coordinates.Opcode); + foreach (var component in coordinates.Operands.Skip(2)) + { + Assert.Equal(usesRenderScale ? SpirvOp.ConvertFToS : SpirvOp.Bitcast, Definition(instructions, component).Opcode); + } + + if (usesRenderScale) + { + // The factor is selected per image, so an unscaled binding costs only the select. + Assert.Contains(instructions, item => item.Opcode == SpirvOp.Select); + Assert.Contains(instructions, item => item.Opcode == SpirvOp.ConvertSToF); + } + else + { + Assert.DoesNotContain(instructions, item => item.Opcode == SpirvOp.ConvertSToF); + } + ValidateWhenAvailable(spirv); + } + + [Theory] + [InlineData(true)] + [InlineData(false)] + public void SizeQueriesReportGuestSizes(bool usesRenderScale) + { + var spirv = CompileImageAccess("ImageGetResinfo", usesRenderScale); + var instructions = Parse(spirv); + var query = Assert.Single(instructions, item => item.Opcode == SpirvOp.ImageQuerySizeLod); + var extracts = instructions + .Where(item => item.Opcode == SpirvOp.CompositeExtract && item.Operands[2] == query.Operands[1]) + .ToArray(); + + Assert.NotEmpty(extracts); + foreach (var extract in extracts) + { + Assert.Equal(usesRenderScale, FeedsOpcode(instructions, extract.Operands[1], SpirvOp.ConvertSToF)); + } + + ValidateWhenAvailable(spirv); + } + + [Theory] + [InlineData(true)] + [InlineData(false)] + public void DynamicSampleOffsetsRemainInGuestTexels(bool usesRenderScale) + { + var spirv = CompileImageAccess("ImageSampleLzO", usesRenderScale); + var instructions = Parse(spirv); + var divide = Assert.Single(instructions, item => item.Opcode == SpirvOp.FDiv); + var offset = Definition(instructions, divide.Operands[2]); + Assert.Equal(usesRenderScale ? SpirvOp.CompositeConstruct : SpirvOp.ConvertSToF, offset.Opcode); + if (usesRenderScale) + { + foreach (var component in offset.Operands.Skip(2)) + { + var multiply = Definition(instructions, component); + Assert.Equal(SpirvOp.FMul, multiply.Opcode); + // Each image selects its factor; unscaled bindings select one. + Assert.Equal(SpirvOp.Select, Definition(instructions, multiply.Operands[3]).Opcode); + } + } + ValidateWhenAvailable(spirv); + } + + private static byte[] CompilePixelPosition(bool usesRenderScale) + { + var program = ResourceTestProgram.Program( + ResourceTestProgram.MoveVector(0, 0, 0), + ResourceTestProgram.EndProgram(4)); + var request = BuildRequest(program, ShaderStage.Pixel, usesRenderScale, + PositionXBit | PositionYBit | PositionZBit); + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + return shader.Spirv; + } + + private static byte[] CompileImageAccess(string opcode, bool usesRenderScale) + { + var program = ResourceTestProgram.Program( + ResourceTestProgram.Image(0, opcode, resourceRegister: 8, dimension: 1, dmask: 0x3), + ResourceTestProgram.EndProgram(8)); + var request = BuildRequest(program, ShaderStage.Compute, usesRenderScale); + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + return shader.Spirv; + } + + private static ShaderCompileRequest BuildRequest(Gen5ShaderProgram program, ShaderStage stage, bool usesRenderScale, + uint pixelInputs = 0) + { + var plan = ShaderResourcePlan.Extract(program, stage, ResourceTestProgram.Hash, 0, 64); + var resources = ResourceMaterializer.ApplyTo(plan, ResourceSpecialization.Default(plan.Info)); + var layout = BindingLayout.Allocate( + resources.Info, + BindingLayout.CollectUserDataRegisters(program, 0, 64), + BindingLayout.UsesGlobalDataShare(program), + ShaderCompileRequest.RequiresFlattenedTable(plan, resources), + BindingLayout.ReadsShaderBase(program), + 0, + false, + usesRenderScale); + return new ShaderCompileRequest(plan, resources, layout) + { + PixelOutputs = stage == ShaderStage.Pixel ? [new Gen5PixelOutputBinding(0, 0, Gen5PixelOutputKind.Float)] : [], + PixelInputAddress = pixelInputs, + PixelInputEnable = pixelInputs, + }; + } + + private static uint FragCoordVariable(IReadOnlyList instructions) => Assert.Single(instructions, + item => item.Opcode == SpirvOp.Decorate && item.Operands.Length >= 3 && + item.Operands[1] == (uint)SpirvDecoration.BuiltIn && item.Operands[2] == (uint)SpirvBuiltIn.FragCoord).Operands[0]; + + private static uint Load(IReadOnlyList instructions, uint variable) => + Assert.Single(instructions, item => item.Opcode == SpirvOp.Load && item.Operands[2] == variable).Operands[1]; + + // True when the given component of the composite is extracted and consumed by the opcode. + private static bool FeedsOperation(IReadOnlyList instructions, uint composite, uint component, SpirvOp opcode) => + instructions.Any(item => item.Opcode == SpirvOp.CompositeExtract && item.Operands[2] == composite && + item.Operands.Length >= 4 && item.Operands[3] == component && + FeedsOpcode(instructions, item.Operands[1], opcode)); + + private static bool FeedsOpcode(IReadOnlyList instructions, uint value, SpirvOp opcode) => + instructions.Any(item => item.Opcode == opcode && item.Operands.Skip(2).Contains(value)); + + // The producer of a value, among the opcodes these tests expect to see one of. + private static Parsed Definition(IReadOnlyList instructions, uint id) => + Assert.Single(instructions, item => ValueOpcodes.Contains(item.Opcode) && item.Operands.Length >= 2 && item.Operands[1] == id); + + private static readonly SpirvOp[] ValueOpcodes = + [ + SpirvOp.CompositeConstruct, SpirvOp.CompositeExtract, SpirvOp.Bitcast, SpirvOp.Load, + SpirvOp.ConvertFToS, SpirvOp.ConvertSToF, SpirvOp.FMul, SpirvOp.FAdd, SpirvOp.Select, + ]; + + private readonly record struct Parsed(SpirvOp Opcode, uint[] Operands); + + private static IReadOnlyList Parse(byte[] spirv) + { + var instructions = new List(); + for (var offset = 5 * sizeof(uint); offset < spirv.Length;) + { + var header = BinaryPrimitives.ReadUInt32LittleEndian(spirv.AsSpan(offset)); + var wordCount = checked((int)(header >> 16)); + Assert.InRange(wordCount, 1, (spirv.Length - offset) / sizeof(uint)); + var operands = new uint[wordCount - 1]; + for (var operand = 0; operand < operands.Length; operand++) + { + operands[operand] = BinaryPrimitives.ReadUInt32LittleEndian(spirv.AsSpan(offset + (operand + 1) * sizeof(uint))); + } + + instructions.Add(new Parsed((SpirvOp)(ushort)header, operands)); + offset += wordCount * sizeof(uint); + } + + return instructions; + } + + private static void ValidateWhenAvailable(byte[] spirv) + { + var sdk = Environment.GetEnvironmentVariable("VULKAN_SDK"); + if (string.IsNullOrEmpty(sdk)) + { + return; + } + + var validator = Path.Combine(sdk, "bin", "spirv-val"); + if (!File.Exists(validator)) + { + return; + } + + var path = Path.GetTempFileName(); + try + { + File.WriteAllBytes(path, spirv); + using var process = Process.Start(new ProcessStartInfo(validator, ["--target-env", "vulkan1.2", path]) + { + RedirectStandardError = true, + })!; + var errors = process.StandardError.ReadToEnd(); + process.WaitForExit(); + Assert.True(process.ExitCode == 0, errors); + } + finally + { + File.Delete(path); + } + } +} diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Gen5ScalarAbsoluteTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Gen5ScalarAbsoluteTests.cs index 61ec628..4e988dd 100644 --- a/tests/SharpEmu.ShaderCompiler.Tests/Gen5ScalarAbsoluteTests.cs +++ b/tests/SharpEmu.ShaderCompiler.Tests/Gen5ScalarAbsoluteTests.cs @@ -25,12 +25,15 @@ public sealed class Gen5ScalarAbsoluteTests Decode(0xBEEA2D6A) with { Pc = 4 }, Decode(0x8F38806Au | (shift << 8)) with { Pc = 8 }, Decode(0x880B380B) with { Pc = 12 }, sample, EndProgram(28)); + var plan = Extract(program, userDataCount: 16); if (!accepted) { - Assert.Throws(() => Extract(program, userDataCount: 16)); + // Lane bits outside the reserved field are part of the sampler: it is read at run time. + Assert.True(plan.Info.UsesRuntimeDescriptors); + Assert.Empty(plan.Info.Samplers); return; } - var plan = Extract(program, userDataCount: 16); + var registers = new uint[16]; registers[8] = 6; // Border clamp keeps the actual border fields live. registers[11] = 0xC0000123; @@ -55,11 +58,60 @@ public sealed class Gen5ScalarAbsoluteTests Branch(44, "SCbranchScc1", -7), EndProgram(48)); var plan = Extract(program, userDataCount: 16); + Assert.Empty(plan.Info.NullDescriptorFallbacks); + if (changesMask) + { + // A loop-carried sampler has no plan-time source: it is read at run time. + Assert.True(plan.Info.UsesRuntimeDescriptors); + return; + } + var registers = new uint[16]; registers[8] = 6; registers[15] = 0xC0000123; Assert.True(RuntimeValueEvaluator.EvaluateDescriptorSource(plan, plan.Info.Samplers[0].Source, Inputs(registers), out var result)); - Assert.Equal(changesMask ? 0u : registers[15], result.Dwords[3]); + Assert.Equal(registers[15], result.Dwords[3]); + } + + // A plain sampled float access whose sampler is loop-carried reads its descriptors at run + // time, as the hardware does, instead of reading the null descriptor. + [Fact] + public void LoopCarriedSamplerOfAPlainSampleIsReadAtRuntime() + { + var sample = Image(28, "ImageSample", 0, 8); + var program = Program( + Decode(0x7D840A81), Decode(0xBEEA2D6A) with { Pc = 4 }, + Decode(0x8F388C6A) with { Pc = 8 }, MoveScalar(12, 24, 0), + Decode(0xBE8B030F) with { Pc = 20 }, Decode(0x880B380B) with { Pc = 24 }, sample, + Sop2(36, "SAddI32", 56, Gen5Operand.Scalar(56), Operand(1)), Sopc(40, "SCmpLgU32", Gen5Operand.Scalar(24), Operand(4)), + Branch(44, "SCbranchScc1", -7), EndProgram(48)); + + var plan = Extract(program, userDataCount: 16); + + Assert.Empty(plan.Info.NullDescriptorFallbacks); + Assert.True(plan.Info.UsesRuntimeDescriptors); + Assert.Contains(plan.Memory.Entries, entry => entry.RuntimeDescriptor); + } + + // Like the sampler case above, an image descriptor carried around the loop has no + // plan-time source. ImageLoad must use the runtime descriptor table rather than make + // the old null-descriptor fallback (which reads zero texels). + [Fact] + public void LoopCarriedImageLoadIsReadAtRuntime() + { + var load = Image(28, "ImageLoad", 8, vectorAddress: 0, dmask: 1); + var program = Program( + Decode(0x7D840A81), Decode(0xBEEA2D6A) with { Pc = 4 }, + Decode(0x8F388C6A) with { Pc = 8 }, MoveScalar(12, 24, 0), + Decode(0xBE8B030F) with { Pc = 20 }, Decode(0x880B380B) with { Pc = 24 }, load, + Sop2(36, "SAddI32", 56, Gen5Operand.Scalar(56), Operand(1)), Sopc(40, "SCmpLgU32", Gen5Operand.Scalar(24), Operand(4)), + Branch(44, "SCbranchScc1", -7), EndProgram(48)); + + var plan = Extract(program, userDataCount: 16); + + Assert.Empty(plan.Info.NullDescriptorFallbacks); + Assert.True(plan.Info.UsesRuntimeDescriptors); + Assert.Contains(plan.Memory.Entries, entry => entry.Pc == 28 && entry.RuntimeDescriptor); } public static TheoryData QuadmaskValues => new() diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Gen5ScalarLaneTransferTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Gen5ScalarLaneTransferTests.cs index 1e4bd1e..3f948d4 100644 --- a/tests/SharpEmu.ShaderCompiler.Tests/Gen5ScalarLaneTransferTests.cs +++ b/tests/SharpEmu.ShaderCompiler.Tests/Gen5ScalarLaneTransferTests.cs @@ -4,6 +4,7 @@ using SharpEmu.HLE; using SharpEmu.ShaderCompiler; using SharpEmu.ShaderCompiler.Vulkan; +using SharpEmu.ShaderCompiler.Metal; using SharpEmu.ShaderCompiler.Resources; using SharpEmu.ShaderCompiler.Tests.Resources; using System.Buffers.Binary; @@ -13,6 +14,45 @@ namespace SharpEmu.ShaderCompiler.Tests; public sealed class Gen5ScalarLaneTransferTests { + [Fact] + public void FlbitOpcodeDecodesAndCompilesWithRdnaSemantics() + { + const ulong shaderAddress = 0x1000; + var memory = new TestCpuMemory(shaderAddress, 0x20); + Span words = stackalloc byte[8]; + BinaryPrimitives.WriteUInt32LittleEndian(words, 0xBEEA156A); // s_flbit_i32_b32 s106, s106 + BinaryPrimitives.WriteUInt32LittleEndian(words[4..], 0xBF810000); // s_endpgm + Assert.True(memory.TryWrite(shaderAddress, words)); + Assert.True(Gen5ShaderTranslator.TryDecodeProgram( + new CpuContext(memory, Generation.Gen5), shaderAddress, + out var program, out var decodeError), decodeError); + Assert.Equal("SFlbitI32B32", program.Instructions[0].Opcode); + var request = ResourceTestProgram.Request(program); + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out _, out var compileError), compileError); + Assert.True(Gen5MslTranslator.TryCompileProgram(request, out _, out var metalError), metalError); + } + + [Theory] + [InlineData(0u, 0xFFFFFFFFu)] + [InlineData(0x0000CCCCu, 16u)] + [InlineData(0xFFFF3333u, 0u)] + [InlineData(0x7FFFFFFFu, 1u)] + [InlineData(0x80000000u, 0u)] + [InlineData(0xFFFFFFFFu, 0u)] + public void FlbitCountsZerosFromTheMostSignificantBit(uint input, uint expected) + { + var program = ResourceTestProgram.Program( + ResourceTestProgram.MoveScalar(0, 106, input), + ResourceTestProgram.Sop1(4, "SFlbitI32B32", 107, Gen5Operand.Scalar(106)), + ResourceTestProgram.MoveScalar(8, 108, 0), + ResourceTestProgram.ScalarLoad(12, 107, 109), + ResourceTestProgram.EndProgram(20)); + var plan = ResourceTestProgram.Extract(program, userDataCount: 0); + var evaluator = new RuntimeValueEvaluator(plan, ResourceTestProgram.Inputs([])); + Assert.True(evaluator.Evaluate(plan.Accesses[0]!.Handle!.Operands[0], out var actual)); + Assert.Equal(expected, actual); + } + [Theory] [InlineData(true)] [InlineData(false)] @@ -322,6 +362,56 @@ public sealed class Gen5ScalarLaneTransferTests Assert.Equal(expectSlot, text.Contains("v86_lane3", StringComparison.Ordinal)); } + [Theory] + [InlineData(false, true)] + [InlineData(true, false)] + public void EmulatedWave64KeepsPureScalarSpillsWithoutAnExchange(bool vectorWrite, bool expectSlot) + { + // Wave64 on a 32-lane host reads another lane through LDS and a workgroup barrier. A VGPR + // that only V_WRITELANE writes holds wave-uniform spilled scalars, so its constant-lane + // reads need no exchange; once an ordinary instruction writes it, the exchange returns. + const ulong shaderAddress = 0x1000; + var memory = new TestCpuMemory(shaderAddress, 0x100); + var words = new List + { + 0xD7610056, 0x0001047E, // v_writelane_b32 v86, s126, 2 + 0xD7610056, 0x0001067F, // v_writelane_b32 v86, s127, 3 + }; + if (vectorWrite) + { + words.Add(0x7EAC0280); // v_mov_b32 v86, 0 + } + + words.AddRange( + [ + 0xD7600000, 0x00010556, // v_readlane_b32 s0, v86, 2 + 0xD7600001, 0x00010756, // v_readlane_b32 s1, v86, 3 + 0xBF810000, // s_endpgm + ]); + var shader = new byte[words.Count * sizeof(uint)]; + for (var index = 0; index < words.Count; index++) + { + BinaryPrimitives.WriteUInt32LittleEndian(shader.AsSpan(index * sizeof(uint)), words[index]); + } + + Assert.True(memory.TryWrite(shaderAddress, shader)); + var ctx = new CpuContext(memory, Generation.Gen5); + Assert.True(Gen5ShaderTranslator.TryDecodeProgram(ctx, shaderAddress, out var program, out var decodeError), decodeError); + var (plan, resources, layout) = ResourceTestProgram.Prepare(program); + var request = new ShaderCompileRequest(plan, resources, layout) + { + LocalSizeX = 64, + ThreadCountX = 64, + WaveSize = 64, + }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var compiled, out var compileError), compileError); + + var text = System.Text.Encoding.ASCII.GetString(compiled.Spirv); + Assert.Equal(expectSlot, text.Contains("v86_lane2", StringComparison.Ordinal)); + Assert.Equal(expectSlot, text.Contains("v86_lane3", StringComparison.Ordinal)); + Assert.Equal(!expectSlot, ReadSpirvOpcodes(compiled.Spirv).Contains((ushort)SpirvOp.ControlBarrier)); + } + private static Gen5ShaderInstruction ScalarInstruction( uint pc, Gen5ShaderEncoding encoding, diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Gen5StructuredControlFlowTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Gen5StructuredControlFlowTests.cs new file mode 100644 index 0000000..52dfca2 --- /dev/null +++ b/tests/SharpEmu.ShaderCompiler.Tests/Gen5StructuredControlFlowTests.cs @@ -0,0 +1,350 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.ShaderCompiler.Vulkan; +using System.Runtime.InteropServices; +using System.Text; +using Xunit; +using static SharpEmu.ShaderCompiler.Tests.Resources.ResourceTestProgram; + +namespace SharpEmu.ShaderCompiler.Tests; + +// Programs with scalar control flow, shared by the translation tests here and the device tests that +// run them through both control-flow forms. Inputs are s8 and s9; results are stored from v4/v5 +// through the buffer V# in s4..s7. Every instruction takes one 4-byte slot. +public sealed class Gen5StructuredControlFlowTests +{ + [Theory] + [InlineData(false)] + [InlineData(true)] + public void RegisterFiles_BelongToTheFunctionExecutingTheProgram(bool forceDispatcher) + { + var code = Compile(NestedIfSharedMerge(), forceDispatcher); + var words = MemoryMarshal.Cast(code.AsSpan()).ToArray(); + var names = new Dictionary(); + var variables = new Dictionary(); + for (var index = 5; index < words.Length; index += (int)(words[index] >> 16)) + { + var length = (int)(words[index] >> 16); + var opcode = (SpirvOp)(words[index] & 0xFFFF); + if (opcode == SpirvOp.Name) + names[words[index + 1]] = Encoding.UTF8.GetString(code.AsSpan((index + 2) * 4, (length - 2) * 4)).TrimEnd('\0'); + if (opcode == SpirvOp.Variable) variables[words[index + 2]] = words[index + 3]; + } + foreach (var prefix in new[] { "s", "v" }) + { + // The compiler now keeps a variable per touched register rather than one array. + var registers = names.Where(entry => entry.Value.StartsWith(prefix, StringComparison.Ordinal) && + uint.TryParse(entry.Value.AsSpan(1), out _)).ToArray(); + Assert.NotEmpty(registers); + foreach (var register in registers) + Assert.Equal((uint)SpirvStorageClass.Function, variables[register.Key]); + } + Gen5LargeDispatcherValidationTests.ValidateWithSpirvToolsWhenAvailable(code); + } + + private static Gen5Operand S(uint register) => Gen5Operand.Scalar(register); + + private static Gen5ShaderInstruction Store(uint pc, uint vector, int offset) => + BufferAccess(pc, "BufferStoreDword", 4, offset: offset, vectorData: vector); + + // for (i = 0; i < s8; i++) acc += (i & 1) != 0 ? i : 100; stores acc, i. + public static Gen5ShaderProgram LoopWithIfElse() => Program( + MoveScalar(0, 10, 0), + MoveScalar(4, 11, 0), + Sopc(8, "SCmpLtU32", S(11), S(8)), + Branch(12, "SCbranchScc0", 7), + Sop2(16, "SAndB32", 12, S(11), Operand(1)), + Branch(20, "SCbranchScc0", 2), + Sop2(24, "SAddU32", 10, S(10), S(11)), + Branch(28, "SBranch", 1), + Sop2(32, "SAddU32", 10, S(10), Operand(100)), + Sop2(36, "SAddU32", 11, S(11), Operand(1)), + Branch(40, "SBranch", -9), + MoveVectorFromScalar(44, 4, 10), + MoveVectorFromScalar(48, 5, 11), + Store(52, 4, 0), + Store(56, 5, 4), + EndProgram(60)); + + public static (uint Acc, uint Second) LoopWithIfElseExpected(uint n, uint unused) + { + uint acc = 0; + uint i = 0; + for (; i < n; i++) + { + acc += (i & 1) != 0 ? i : 100u; + } + + return (acc, i); + } + + // Nested loops that leave the program early: + // for (i < s8) for (j < i) { acc += j; if (acc == s9) { store acc, 0xDEAD; end } } stores acc, i. + public static Gen5ShaderProgram NestedLoopsWithEarlyExit() => Program( + MoveScalar(0, 10, 0), + MoveScalar(4, 11, 0), + Sopc(8, "SCmpLtU32", S(11), S(8)), + Branch(12, "SCbranchScc0", 10), + MoveScalar(16, 12, 0), + Sopc(20, "SCmpLtU32", S(12), S(11)), + Branch(24, "SCbranchScc0", 5), + Sop2(28, "SAddU32", 10, S(10), S(12)), + Sopc(32, "SCmpEqU32", S(10), S(9)), + Branch(36, "SCbranchScc1", 9), + Sop2(40, "SAddU32", 12, S(12), Operand(1)), + Branch(44, "SBranch", -7), + Sop2(48, "SAddU32", 11, S(11), Operand(1)), + Branch(52, "SBranch", -12), + MoveVectorFromScalar(56, 4, 10), + MoveVectorFromScalar(60, 5, 11), + Store(64, 4, 0), + Store(68, 5, 4), + EndProgram(72), + MoveVectorFromScalar(76, 4, 10), + MoveVector(80, 5, 0xDEAD), + Store(84, 4, 0), + Store(88, 5, 4), + EndProgram(92)); + + public static (uint Acc, uint Second) NestedLoopsWithEarlyExitExpected(uint n, uint target) + { + uint acc = 0; + for (uint i = 0; i < n; i++) + { + for (uint j = 0; j < i; j++) + { + acc += j; + if (acc == target) + { + return (acc, 0xDEAD); + } + } + } + + return (acc, n); + } + + // if (s8 & 1) { acc += 1; if (s8 & 2) acc += 10; } acc += 100; the inner selection merges where + // the outer one does. + public static Gen5ShaderProgram NestedIfSharedMerge() => Program( + MoveScalar(0, 10, 0), + Sop2(4, "SAndB32", 12, S(8), Operand(1)), + Branch(8, "SCbranchScc0", 4), + Sop2(12, "SAddU32", 10, S(10), Operand(1)), + Sop2(16, "SAndB32", 12, S(8), Operand(2)), + Branch(20, "SCbranchScc0", 1), + Sop2(24, "SAddU32", 10, S(10), Operand(10)), + Sop2(28, "SAddU32", 10, S(10), Operand(100)), + MoveVectorFromScalar(32, 4, 10), + MoveVector(36, 5, 0), + Store(40, 4, 0), + Store(44, 5, 4), + EndProgram(48)); + + public static (uint Acc, uint Second) NestedIfSharedMergeExpected(uint n, uint unused) + { + uint acc = 0; + if ((n & 1) != 0) + { + acc += 1; + if ((n & 2) != 0) + { + acc += 10; + } + } + + return (acc + 100, 0); + } + + // Two entries into one cycle (A <-> B), so the graph is irreducible and keeps the dispatcher. + public static Gen5ShaderProgram IrreducibleCycle() => Program( + MoveScalar(0, 10, 0), + Sopc(4, "SCmpEqU32", S(8), Operand(0)), + Branch(8, "SCbranchScc1", 3), + Sop2(12, "SAddU32", 10, S(10), Operand(1)), + Sopc(16, "SCmpLtU32", S(10), S(9)), + Branch(20, "SCbranchScc0", 3), + Sop2(24, "SAddU32", 10, S(10), Operand(2)), + Sopc(28, "SCmpLtU32", S(10), S(9)), + Branch(32, "SCbranchScc1", -6), + MoveVectorFromScalar(36, 4, 10), + MoveVector(40, 5, 0), + Store(44, 4, 0), + Store(48, 5, 4), + EndProgram(52)); + + public static (uint Acc, uint Second) IrreducibleCycleExpected(uint n, uint limit) + { + uint acc = 0; + var inB = n == 0; + while (true) + { + if (!inB) + { + acc += 1; + if (!(acc < limit)) + { + break; + } + } + + inB = false; + acc += 2; + if (!(acc < limit)) + { + break; + } + } + + return (acc, 0); + } + + public static IEnumerable StructuredPrograms() + { + yield return [nameof(LoopWithIfElse)]; + yield return [nameof(NestedLoopsWithEarlyExit)]; + yield return [nameof(NestedIfSharedMerge)]; + } + + public static Gen5ShaderProgram Create(string name) => name switch + { + nameof(LoopWithIfElse) => LoopWithIfElse(), + nameof(NestedLoopsWithEarlyExit) => NestedLoopsWithEarlyExit(), + nameof(NestedIfSharedMerge) => NestedIfSharedMerge(), + nameof(IrreducibleCycle) => IrreducibleCycle(), + _ => throw new ArgumentOutOfRangeException(nameof(name)), + }; + + public static (uint Acc, uint Second) Expected(string name, uint input0, uint input1) => name switch + { + nameof(LoopWithIfElse) => LoopWithIfElseExpected(input0, input1), + nameof(NestedLoopsWithEarlyExit) => NestedLoopsWithEarlyExitExpected(input0, input1), + nameof(NestedIfSharedMerge) => NestedIfSharedMergeExpected(input0, input1), + nameof(IrreducibleCycle) => IrreducibleCycleExpected(input0, input1), + _ => throw new ArgumentOutOfRangeException(nameof(name)), + }; + + [Theory] + [MemberData(nameof(StructuredPrograms))] + public void StructurableProgramIsEmittedWithoutTheDispatcher(string name) + { + var structured = Compile(Create(name), forceDispatcher: false); + var dispatcher = Compile(Create(name), forceDispatcher: true); + + // The structured body is its own function, called once from the entry point. + Assert.Contains(SpirvOp.FunctionCall, Gen5LargeDispatcherValidationTests.ReadSpirvOpcodes(structured)); + Assert.DoesNotContain(SpirvOp.FunctionCall, Gen5LargeDispatcherValidationTests.ReadSpirvOpcodes(dispatcher)); + foreach (var code in new[] { structured, dispatcher }) + { + Gen5LargeDispatcherValidationTests.ValidateStructuredControlFlow(code); + Gen5LargeDispatcherValidationTests.ValidateWithSpirvToolsWhenAvailable(code); + } + } + + [Fact] + public void IrreducibleProgramKeepsTheDispatcher() + { + var code = Compile(IrreducibleCycle(), forceDispatcher: false); + Assert.DoesNotContain(SpirvOp.FunctionCall, Gen5LargeDispatcherValidationTests.ReadSpirvOpcodes(code)); + Gen5LargeDispatcherValidationTests.ValidateStructuredControlFlow(code); + Gen5LargeDispatcherValidationTests.ValidateWithSpirvToolsWhenAvailable(code); + } + + [Fact] + public void LoopsAreDeclaredWithLoopMerges() + { + var opcodes = Gen5LargeDispatcherValidationTests.ReadSpirvOpcodes(Compile(NestedLoopsWithEarlyExit(), forceDispatcher: false)); + Assert.Equal(2, opcodes.Count(static opcode => opcode == SpirvOp.LoopMerge)); + } + + // A loop that can leave through a chain of diamonds, each of whose sides flows into the next. + // Copying that exit tail at the loop exit doubles at every diamond. + public static Gen5ShaderProgram LoopExitThroughDiamondChain(int diamonds) + { + var instructions = new List + { + MoveScalar(0, 10, 0), + MoveScalar(4, 11, 0), + Sop2(8, "SAddU32", 11, S(11), Operand(1)), + Sopc(12, "SCmpEqU32", S(11), S(9)), + Branch(16, "SCbranchScc1", 3), // -> 32, the diamond chain + Sopc(20, "SCmpLtU32", S(11), S(8)), + Branch(24, "SCbranchScc1", -5), // -> 8 + EndProgram(28), + }; + uint pc = 32; + for (var diamond = 0; diamond < diamonds; diamond++) + { + instructions.Add(Sopc(pc, "SCmpEqU32", S(11), Operand((uint)diamond))); + instructions.Add(Branch(pc + 4, "SCbranchScc1", 2)); + instructions.Add(Sop2(pc + 8, "SAddU32", 10, S(10), Operand(1))); + instructions.Add(Branch(pc + 12, "SBranch", 1)); + instructions.Add(Sop2(pc + 16, "SAddU32", 10, S(10), Operand(2))); + pc += 20; + } + + instructions.Add(MoveVectorFromScalar(pc, 4, 10)); + instructions.Add(Store(pc + 4, 4, 0)); + instructions.Add(EndProgram(pc + 8)); + return Program([.. instructions]); + } + + [Fact] + public void ExitTailCopiesStayBounded() + { + var structured = Compile(LoopExitThroughDiamondChain(14), forceDispatcher: false); + var dispatcher = Compile(LoopExitThroughDiamondChain(14), forceDispatcher: true); + Assert.True( + structured.Length <= dispatcher.Length * 4, + $"structured={structured.Length} bytes, dispatcher={dispatcher.Length} bytes"); + Gen5LargeDispatcherValidationTests.ValidateStructuredControlFlow(structured); + Gen5LargeDispatcherValidationTests.ValidateWithSpirvToolsWhenAvailable(structured); + } + + // The control-flow skeleton of a Yotei compute shader: early execz exits to the final + // S_ENDPGM before and after a loop, and vccnz skips over the loop. + public static Gen5ShaderProgram EarlyExitsAroundLoop() => Program( + Nop(0), Branch(4, "SCbranchExecz", 22), Nop(8), Branch(12, "SCbranchExecz", 20), Nop(16), + Branch(20, "SCbranchVccnz", 10), Nop(24), Branch(28, "SCbranchVccnz", 8), Nop(32), Nop(36), + Branch(40, "SCbranchVccz", 3), Nop(44), Branch(48, "SBranch", -4), Nop(52), Nop(56), Nop(60), + Branch(64, "SCbranchExecz", 7), Nop(68), Branch(72, "SCbranchExecz", 3), Nop(76), Nop(80), + Nop(84), Nop(88), Nop(92), EndProgram(96)); + + [Fact] + public void EarlyExitsAroundLoopAreStructured() + { + var code = Compile(EarlyExitsAroundLoop(), forceDispatcher: false); + Assert.Contains(SpirvOp.FunctionCall, Gen5LargeDispatcherValidationTests.ReadSpirvOpcodes(code)); + Gen5LargeDispatcherValidationTests.ValidateStructuredControlFlow(code); + Gen5LargeDispatcherValidationTests.ValidateWithSpirvToolsWhenAvailable(code); + } + + // One arm reaches the shared block directly; the other either reaches it or ends the program + // on its own, so the shared block post-dominates nothing. + public static Gen5ShaderProgram EarlyReturnBesideSharedBlock() => Program( + Nop(0), Branch(4, "SCbranchScc1", 5), Nop(8), Branch(12, "SCbranchScc1", 1), + Branch(16, "SBranch", 2), Nop(20), EndProgram(24), Nop(28), EndProgram(32)); + + [Fact] + public void EarlyReturnBesideSharedBlockIsStructured() + { + var code = Compile(EarlyReturnBesideSharedBlock(), forceDispatcher: false); + Assert.Contains(SpirvOp.FunctionCall, Gen5LargeDispatcherValidationTests.ReadSpirvOpcodes(code)); + Gen5LargeDispatcherValidationTests.ValidateStructuredControlFlow(code); + Gen5LargeDispatcherValidationTests.ValidateWithSpirvToolsWhenAvailable(code); + } + + internal static byte[] Compile(Gen5ShaderProgram program, bool forceDispatcher) + { + var (plan, resources, layout) = Prepare(program); + var request = new ShaderCompileRequest(plan, resources, layout) + { + LocalSizeX = 1, + ThreadCountX = 1, + ForceDispatcher = forceDispatcher, + }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + return shader.Spirv; + } +} diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Gen5TessellationTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Gen5TessellationTests.cs new file mode 100644 index 0000000..16d69e6 --- /dev/null +++ b/tests/SharpEmu.ShaderCompiler.Tests/Gen5TessellationTests.cs @@ -0,0 +1,230 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Buffers.Binary; +using SharpEmu.ShaderCompiler; +using SharpEmu.ShaderCompiler.Resources; +using SharpEmu.ShaderCompiler.Tests.Resources; +using SharpEmu.ShaderCompiler.Vulkan; +using Xunit; +using static SharpEmu.ShaderCompiler.Tests.Resources.ResourceTestProgram; + +namespace SharpEmu.ShaderCompiler.Tests; + +public sealed class Gen5TessellationTests +{ + [Theory] + [InlineData(6u)] + [InlineData(32u)] + public void MergedHull_UsesSeparateBackUserDataAfterLocalRegistersAreReused(uint frontCount) + { + // LS receives an inline descriptor and its own table. HS receives a + // separate table containing both that LS table and the ring table. + ulong frontTable = 0x20_0010_0000; + ulong backTable = frontTable + 0x1000; + ulong rings = frontTable + 0x2000; + var memory = new TestWordMemory { Base = frontTable, Words = new uint[4096] }; + memory.At(frontTable) = 0x6000; + memory.At(frontTable + 4) = 0; + memory.At(frontTable + 8) = 128; + memory.At(frontTable + 12) = 0; + memory.At(backTable) = (uint)frontTable; + memory.At(backTable + 4) = (uint)(frontTable >> 32); + memory.At(backTable + 8) = (uint)rings; + memory.At(backTable + 12) = (uint)(rings >> 32); + memory.At(rings + 48) = 0x7000; + memory.At(rings + 52) = 0; + memory.At(rings + 56) = 256; + memory.At(rings + 60) = 0; + var program = Program( + ScalarLoad(0, 8, 16, 4), BufferLoad(8, 16), + MoveScalar(16, 8, 0xBAD), MoveScalar(20, 9, 0xBAD), + ScalarLoad(256, 0, 8, 4), ScalarLoad(264, 10, 36, 4, 48), + BufferStore(272, 36), EndProgram(280)) with { FusedContinuationPc = 256 }; + var lowered = Gen5TessellationLowering.PrepareMergedHull(program, 8 + frontCount); + var data = new uint[frontCount + 2]; + data[0] = (uint)frontTable; + data[1] = (uint)(frontTable >> 32); + data[frontCount] = (uint)backTable; + data[frontCount + 1] = (uint)(backTable >> 32); + var plan = ShaderResourcePlan.Extract(lowered, ShaderStage.Compute, Hash, 8, (uint)data.Length, waveSize: 64); + var snapshot = new ResourceSnapshot(); + var specialization = new ResourceSpecialization(); + Assert.True(ResourceMaterializer.Materialize(plan, Inputs(data, memory.Read), ref snapshot, ref specialization)); + Assert.Equal(0x6000u, snapshot.Buffers[0][0]); + Assert.Equal(0x7000u, snapshot.Buffers[1][0]); + } + + [Fact] + public void FactorBuffer_ResolvesHardwareOffsetStoresAndRejectsAmbiguousOrReusedOffsets() + { + Gen5ShaderInstruction Store(uint pc, uint descriptor) + { + var instruction = BufferAccess(pc, "BufferStoreDword", descriptor); + return instruction with { Sources = [..instruction.Sources.Take(2), Gen5Operand.Scalar(4)] }; + } + ShaderResourcePlan Plan(params Gen5ShaderInstruction[] instructions) => + ShaderResourcePlan.Extract(Program(instructions), ShaderStage.Compute, Hash, 8, 24, waveSize: 64); + var plan = Plan(Store(0, 12), Store(8, 12), EndProgram(16)); + Assert.Equal(plan.Memory.Find(0)!.Resource, Gen5TessellationLowering.FindFactorBuffer(plan, 0)); + Assert.Throws(() => Gen5TessellationLowering.FindFactorBuffer( + Plan(Store(0, 12), Store(8, 16), EndProgram(16)), 0)); + Assert.Throws(() => Gen5TessellationLowering.FindFactorBuffer( + Plan(MoveScalar(0, 4, 0), Store(4, 12), EndProgram(12)), 0)); + Assert.Throws(() => Gen5TessellationLowering.FindFactorBuffer(Plan(EndProgram(0)), 0)); + } + + [Theory] + [InlineData(Gen5TessellationDomain.Quads, 6)] + [InlineData(Gen5TessellationDomain.Triangles, 4)] + [InlineData(Gen5TessellationDomain.Isolines, 2)] + public void ControlBridge_UsesGuestFactorBufferAndNativePatchOutputs(Gen5TessellationDomain domain, int factorCount) + { + var (plan, resources, _) = Prepare(Program(EndProgram(0)), ShaderStage.TessellationEvaluation); + var layout = BindingLayout.Allocate(resources.Info, [], false, false, false, usesTessellationData: true); + Assert.NotNull(layout.Find(DescriptorBindingKind.ShaderData)); + var shader = Gen5TessellationBridge.CompileControl(layout, domain); + Gen5LargeDispatcherValidationTests.ValidateWithSpirvToolsWhenAvailable(shader.Spirv); + var instructions = Instructions(shader.Spirv).ToArray(); + Assert.Contains(instructions, i => i.Op == SpirvOp.EntryPoint && i.Args[0] == 1); + Assert.Contains(instructions, i => i.Op == SpirvOp.ExecutionMode && i.Args[1] == 26 && i.Args[2] == 1); + Assert.Equal(factorCount, instructions.Count(i => i.Op == SpirvOp.ConvertUToPtr)); + Assert.Equal(2, instructions.Count(i => i.Op == SpirvOp.Decorate && i.Args[1] == (uint)SpirvDecoration.Patch)); + } + + [Fact] + public void MergedHullLowering_PublishesLocalOutputsAndPreservesOriginalAddresses() + { + var front = Program(Nop(0), Nop(4), Nop(8), EndProgram(256)) with { FusedContinuationPc = 256 }; + var lowered = Gen5TessellationLowering.PrepareMergedHull(front, 14); + Assert.Equal("SMovB64", lowered.Instructions[0].Opcode); + Assert.Equal(Gen5Operand.Scalar(14), lowered.Instructions[0].Sources[0]); + Assert.Equal(Gen5Operand.Scalar(0), lowered.Instructions[0].Destinations[0]); + Assert.Contains(lowered.Instructions, i => i.Pc == 260 && i.Opcode == "SBarrier"); + Assert.Equal(264u, lowered.FusedContinuationPc); + Assert.Equal(256ul, lowered.InstructionAddressOffset(264)); + Assert.Equal(0ul, lowered.InstructionAddressOffset(4)); + Assert.Throws(() => Gen5TessellationLowering.PrepareMergedHull(Program(EndProgram(0)), 14)); + } + + [Theory] + [InlineData(32u, 128u)] + [InlineData(64u, 63u)] + [InlineData(64u, 320u)] + public void CooperativeWaves_RejectIncompatibleExecutionShapes(uint waveSize, uint threads) + { + var (plan, resources, layout) = Prepare(Program(EndProgram(0))); + Assert.False(Gen5SpirvTranslator.TryCompileProgram(new(plan, resources, layout) + { CooperativeWave64Workgroup = true, WaveSize = waveSize, LocalSizeX = threads }, out _, out var error)); + Assert.Contains("Cooperative wave64", error); + } + + [Fact] + public void HullConfiguration_UsesDirectCountsAndSeparateInputAndOutputSizes() + { + Assert.True(Gen5TessellationHullInfo.TryDecode(63 | (4u << 8) | (4u << 14), 1024, out var info, out var error), error); + Assert.Equal(new Gen5TessellationHullInfo(4, 4, 63, 1024), info); + Assert.True(Gen5TessellationHullInfo.TryDecode(8 | (6u << 8) | (12u << 14), 256, out info, out error), error); + Assert.Equal(new Gen5TessellationHullInfo(6, 12, 8, 256), info); + } + + [Theory] + [InlineData(0u)] + [InlineData(1u | (33u << 8) | (4u << 14))] + [InlineData(65u | (4u << 8) | (4u << 14))] + public void HullConfiguration_RejectsUnrepresentableWorkgroups(uint configuration) => + Assert.False(Gen5TessellationHullInfo.TryDecode(configuration, 256, out _, out _)); + + [Theory] + [InlineData(2u | (2u << 2) | (3u << 5), Gen5TessellationDomain.Quads, Gen5TessellationSpacing.FractionalOdd, false, false)] + [InlineData(1u | (3u << 2) | (2u << 5), Gen5TessellationDomain.Triangles, Gen5TessellationSpacing.FractionalEven, false, true)] + [InlineData(1u << 5, Gen5TessellationDomain.Isolines, Gen5TessellationSpacing.Equal, false, false)] + [InlineData(2u, Gen5TessellationDomain.Quads, Gen5TessellationSpacing.Equal, true, false)] + public void Parameters_DecodeIndependentDomainSpacingAndTopology(uint parameter, + Gen5TessellationDomain domain, Gen5TessellationSpacing spacing, bool point, bool clockwise) + { + Assert.True(Gen5TessellationInfo.TryDecode(parameter | (2u << 17), out var info, out var error), error); + Assert.Equal(new Gen5TessellationInfo(domain, spacing, point, clockwise), info); + } + + [Theory] + [InlineData(3u)] + [InlineData(4u << 2)] + [InlineData(4u << 5)] + [InlineData(2u | (1u << 5))] + [InlineData(2u << 5)] + public void Parameters_RejectReservedOrIncompatibleFields(uint parameter) + { + Assert.False(Gen5TessellationInfo.TryDecode(parameter, out _, out var error)); + Assert.Contains("Invalid tessellator", error); + } + + [Theory] + [InlineData(Gen5TessellationDomain.Quads, Gen5TessellationSpacing.FractionalOdd, false, false)] + [InlineData(Gen5TessellationDomain.Triangles, Gen5TessellationSpacing.FractionalEven, false, true)] + [InlineData(Gen5TessellationDomain.Isolines, Gen5TessellationSpacing.Equal, false, false)] + [InlineData(Gen5TessellationDomain.Quads, Gen5TessellationSpacing.Equal, true, false)] + public void Evaluation_UsesNativeTessellatorInputsAndModes(Gen5TessellationDomain domain, + Gen5TessellationSpacing spacing, bool point, bool clockwise) + { + var program = Program(new Gen5ShaderInstruction(0, Gen5ShaderEncoding.Exp, "Exp", [0u, 0u], + [Gen5Operand.Vector(5), Gen5Operand.Vector(6), Gen5Operand.Vector(7), Gen5Operand.Source(242)], [], + new Gen5ExportControl(12, 15, false, true, false)), EndProgram(8)); + var (plan, resources, layout) = Prepare(program, ShaderStage.TessellationEvaluation); + var request = new ShaderCompileRequest(plan, resources, layout) + { Tessellation = new(domain, spacing, point, clockwise) }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + Gen5LargeDispatcherValidationTests.ValidateWithSpirvToolsWhenAvailable(shader.Spirv); + var instructions = Instructions(shader.Spirv).ToArray(); + Assert.Contains(instructions, i => i.Op == SpirvOp.EntryPoint && i.Args[0] == 2); + Assert.Contains(instructions, i => i.Op == SpirvOp.Decorate && i.Args[1] == (uint)SpirvDecoration.BuiltIn && i.Args[2] == 13); + Assert.Contains(instructions, i => i.Op == SpirvOp.Decorate && i.Args[1] == (uint)SpirvDecoration.BuiltIn && i.Args[2] == 7); + var mode = domain switch { Gen5TessellationDomain.Quads => 24u, Gen5TessellationDomain.Triangles => 22u, _ => 25u }; + Assert.Contains(instructions, i => i.Op == SpirvOp.ExecutionMode && i.Args[1] == mode); + Assert.DoesNotContain(instructions, i => i.Op == SpirvOp.ExecutionMode && i.Args[1] == 17); + } + + // The ES ABI gives the domain shader u, v, its relative patch and its patch ID in v5..v8. + [Fact] + public void Evaluation_FillsBothThePatchWithinItsGroupAndThePatchId() + { + var program = Program(new Gen5ShaderInstruction(0, Gen5ShaderEncoding.Exp, "Exp", [0u, 0u], + [Gen5Operand.Vector(7), Gen5Operand.Vector(8), Gen5Operand.Vector(5), Gen5Operand.Vector(6)], [], + new Gen5ExportControl(12, 15, false, true, false)), EndProgram(8)); + var (plan, resources, layout) = Prepare(program, ShaderStage.TessellationEvaluation); + var request = new ShaderCompileRequest(plan, resources, layout) + { Tessellation = new(Gen5TessellationDomain.Quads, Gen5TessellationSpacing.Equal, false, false) }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + Gen5LargeDispatcherValidationTests.ValidateWithSpirvToolsWhenAvailable(shader.Spirv); + var instructions = Instructions(shader.Spirv).ToArray(); + var primitiveId = Assert.Single(instructions, + i => i.Op == SpirvOp.Decorate && i.Args[1] == (uint)SpirvDecoration.BuiltIn && i.Args[2] == 7).Args[0]; + var derived = instructions.Where(i => i.Op == SpirvOp.Load && i.Args[2] == primitiveId).Select(i => i.Args[1]).ToHashSet(); + foreach (var add in instructions.Where(i => i.Op == SpirvOp.IAdd && (derived.Contains(i.Args[2]) || derived.Contains(i.Args[3])))) + derived.Add(add.Args[1]); + var stored = instructions.Where(i => i.Op == SpirvOp.Store && derived.Contains(i.Args[1])).Select(i => i.Args[0]).ToHashSet(); + // Two distinct registers receive the native primitive ID: v7 and v8. + Assert.Equal(2, stored.Count); + } + + [Fact] + public void Evaluation_MissingConfigurationIsAnExplicitFailure() + { + var (plan, resources, layout) = Prepare(Program(EndProgram(0)), ShaderStage.TessellationEvaluation); + Assert.False(Gen5SpirvTranslator.TryCompileProgram(new(plan, resources, layout), out _, out var error)); + Assert.Contains("tessellation", error, StringComparison.OrdinalIgnoreCase); + } + + private static IEnumerable<(SpirvOp Op, uint[] Args)> Instructions(byte[] bytes) + { + for (var offset = 20; offset < bytes.Length;) + { + var header = BinaryPrimitives.ReadUInt32LittleEndian(bytes.AsSpan(offset)); + var count = (int)(header >> 16); + var args = new uint[count - 1]; + for (var i = 0; i < args.Length; i++) args[i] = BinaryPrimitives.ReadUInt32LittleEndian(bytes.AsSpan(offset + 4 * (i + 1))); + yield return ((SpirvOp)(header & 0xFFFF), args); + offset += count * 4; + } + } +} diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Gen5TypedBufferLoadDecodeTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Gen5TypedBufferLoadDecodeTests.cs index 41e1aba..8d8e18a 100644 --- a/tests/SharpEmu.ShaderCompiler.Tests/Gen5TypedBufferLoadDecodeTests.cs +++ b/tests/SharpEmu.ShaderCompiler.Tests/Gen5TypedBufferLoadDecodeTests.cs @@ -10,6 +10,15 @@ namespace SharpEmu.ShaderCompiler.Tests; // Typed buffer instructions keep their own unified format; formatted untyped ones do not carry one. public sealed class Gen5TypedBufferLoadDecodeTests { + [Fact] + public void ComparisonGatherWithExplicitLodDecodesItsOpcodeAndLength() + { + var program = Decode([0xF0000100u | (0x4Cu << 18), 0x00000004u, 0xBF810000u]); + var gather = program.Instructions[0]; + Assert.Equal("ImageGather4CL", gather.Opcode); + Assert.Equal(Gen5ShaderEncoding.Mimg, gather.Encoding); + Assert.Equal(8u, program.Instructions[1].Pc); + } private const ulong ShaderAddress = 0x1000; private const uint Format32x4Float = 77; @@ -68,6 +77,32 @@ public sealed class Gen5TypedBufferLoadDecodeTests Assert.Equal(0u, control.TypedFormat); } + [Theory] + [InlineData(0x80u, "BufferLoadFormatD16X", 1u)] + [InlineData(0x81u, "BufferLoadFormatD16Xy", 1u)] + [InlineData(0x82u, "BufferLoadFormatD16Xyz", 2u)] + [InlineData(0x83u, "BufferLoadFormatD16Xyzw", 2u)] + [InlineData(0x84u, "BufferStoreFormatD16X", 1u)] + [InlineData(0x85u, "BufferStoreFormatD16Xy", 1u)] + [InlineData(0x86u, "BufferStoreFormatD16Xyz", 2u)] + [InlineData(0x87u, "BufferStoreFormatD16Xyzw", 2u)] + public void UntypedD16_UsesOpcodeBit25AndPackedRegisterCount(uint opcode, string expected, uint count) + { + var instruction = Decode([0xE0002000u | (opcode << 18), 0x80011A14u, 0xBF810000u]).Instructions[0]; + Assert.Equal(expected, instruction.Opcode); + Assert.Equal(count, Assert.IsType(instruction.Control).DwordCount); + Assert.Equal((int)count, instruction.Destinations.Count); + Assert.DoesNotContain(instruction.Destinations, destination => destination.Value == 28); + } + + [Fact] + public void OddBufferOffset_DoesNotChangeOpcode() + { + var instruction = Decode([0xE00C2001u, 0x80011A14u, 0xBF810000u]).Instructions[0]; + Assert.Equal("BufferLoadFormatXyzw", instruction.Opcode); + Assert.Equal(1, Assert.IsType(instruction.Control).OffsetBytes); + } + private static Gen5ShaderProgram Decode(uint[] words) { var context = WriteProgram(new TestCpuMemory(ShaderAddress, 0x100), words); diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Gen5WaveMaskLaneFlagTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Gen5WaveMaskLaneFlagTests.cs new file mode 100644 index 0000000..88295cc --- /dev/null +++ b/tests/SharpEmu.ShaderCompiler.Tests/Gen5WaveMaskLaneFlagTests.cs @@ -0,0 +1,126 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Buffers.Binary; +using SharpEmu.HLE; +using SharpEmu.ShaderCompiler; +using SharpEmu.ShaderCompiler.Resources; +using SharpEmu.ShaderCompiler.Tests.Resources; +using SharpEmu.ShaderCompiler.Vulkan; +using Xunit; + +namespace SharpEmu.ShaderCompiler.Tests; + +// VCC and EXEC carry a per-lane boolean that every write to their registers refreshes. +public sealed class Gen5WaveMaskLaneFlagTests +{ + private const ulong ShaderAddress = 0x1_0000_0000; + + [Fact] + public void EmulatedWave64TestsTheLaneBitWithoutComposingTheMask() + { + var program = Decode( + [ + 0x7C020300, // v_cmp_lt_f32 vcc, v0, v1 (declares the lane input and writes a mask) + 0xBEFE0300, // s_mov_b32 exec_lo, s0 + 0xBF810000, // s_endpgm + ]); + Assert.Equal("VCmpLtF32", program.Instructions[0].Opcode); + + var (plan, resources, layout) = ResourceTestProgram.Prepare(program, ShaderStage.Compute, 0, 0, 0); + // A 64-thread group is the wave64-on-32-lanes case the translator emulates. + var request = new ShaderCompileRequest(plan, resources, layout) + { + WaveSize = 64, + LocalSizeX = 8, + LocalSizeY = 8, + LocalSizeZ = 1, + }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + + // The lane's bit lives in one of the two mask registers, so the refresh selects that + // register and shifts inside it. Composing the 64-bit mask and masking it with a 64-bit + // lane bit left 27 of these for this two-instruction program; the remaining six build + // the mask value itself, which stays 64-bit because the guest register pair is. + Assert.Equal(6, Count64BitBitwiseAnd(shader.Spirv)); + } + + private static int Count64BitBitwiseAnd(byte[] spirv) + { + var words = new uint[spirv.Length / 4]; + Buffer.BlockCopy(spirv, 0, words, 0, spirv.Length); + var wideTypes = new HashSet(); + var count = 0; + for (var offset = 5; offset < words.Length;) + { + var wordCount = (int)(words[offset] >> 16); + var opcode = (ushort)(words[offset] & 0xFFFF); + if (opcode == (ushort)SpirvOp.TypeInt && words[offset + 2] == 64) + { + wideTypes.Add(words[offset + 1]); + } + else if (opcode == (ushort)SpirvOp.BitwiseAnd && wideTypes.Contains(words[offset + 1])) + { + count++; + } + + offset += wordCount; + } + + return count; + } + + private static Gen5ShaderProgram Decode(IReadOnlyList words) + { + var memory = new TestCpuMemory(ShaderAddress, words.Count * sizeof(uint)); + var bytes = new byte[words.Count * sizeof(uint)]; + for (var index = 0; index < words.Count; index++) + { + BinaryPrimitives.WriteUInt32LittleEndian(bytes.AsSpan(index * sizeof(uint)), words[index]); + } + + Assert.True(memory.TryWrite(ShaderAddress, bytes)); + var context = new CpuContext(memory, Generation.Gen5); + Assert.True(Gen5ShaderTranslator.TryDecodeProgram(context, ShaderAddress, out var program, out var error), error); + return program; + } + + private sealed class TestCpuMemory(ulong baseAddress, int size) : ICpuMemory + { + private readonly byte[] _storage = new byte[size]; + + public bool TryRead(ulong virtualAddress, Span destination) + { + if (!TryResolve(virtualAddress, destination.Length, out var offset)) + { + return false; + } + + _storage.AsSpan(offset, destination.Length).CopyTo(destination); + return true; + } + + public bool TryWrite(ulong virtualAddress, ReadOnlySpan source) + { + if (!TryResolve(virtualAddress, source.Length, out var offset)) + { + return false; + } + + source.CopyTo(_storage.AsSpan(offset, source.Length)); + return true; + } + + private bool TryResolve(ulong address, int length, out int offset) + { + offset = 0; + if (address < baseAddress || address - baseAddress > int.MaxValue) + { + return false; + } + + offset = (int)(address - baseAddress); + return offset <= _storage.Length - length; + } + } +} diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Resources/BindingLayoutTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Resources/BindingLayoutTests.cs index 165bb08..3a13661 100644 --- a/tests/SharpEmu.ShaderCompiler.Tests/Resources/BindingLayoutTests.cs +++ b/tests/SharpEmu.ShaderCompiler.Tests/Resources/BindingLayoutTests.cs @@ -150,15 +150,17 @@ public sealed class BindingLayoutTests [Fact] public void ImageBindingAbi() { - Assert.Equal(45u, BindingLayout.ImageBindingCount); + Assert.Equal(51u, BindingLayout.ImageBindingCount); Assert.Equal(0u, (uint)DescriptorBindingKind.Buffers); - Assert.Equal(46u, (uint)DescriptorBindingKind.Samplers); - Assert.Equal(47u, (uint)DescriptorBindingKind.GlobalDataShare); - Assert.Equal(48u, (uint)DescriptorBindingKind.DeviceAddressPageTable); - Assert.Equal(49u, (uint)DescriptorBindingKind.FaultBuffer); - Assert.Equal(50u, (uint)DescriptorBindingKind.FlattenedResourceTable); - Assert.Equal(51u, (uint)DescriptorBindingKind.ShaderData); - Assert.Equal(52u, (uint)DescriptorBindingKind.Count); + Assert.Equal(52u, (uint)DescriptorBindingKind.Samplers); + Assert.Equal(53u, (uint)DescriptorBindingKind.GlobalDataShare); + Assert.Equal(54u, (uint)DescriptorBindingKind.DeviceAddressPageTable); + Assert.Equal(55u, (uint)DescriptorBindingKind.FaultBuffer); + Assert.Equal(56u, (uint)DescriptorBindingKind.FlattenedResourceTable); + Assert.Equal(57u, (uint)DescriptorBindingKind.ShaderData); + Assert.Equal(58u, (uint)DescriptorBindingKind.RuntimeDescriptorTable); + Assert.Equal(59u, (uint)DescriptorBindingKind.RuntimeDescriptorMisses); + Assert.Equal(60u, (uint)DescriptorBindingKind.Count); ImageDimension[] sampledDimensions = [ @@ -178,6 +180,10 @@ public sealed class BindingLayoutTests Assert.Equal(BindingLayout.FirstImageBinding + index, (uint)kind!.Value); Assert.Equal(index, ImageDescriptorBinding.ArrayIndex(kind.Value)); Assert.Equal(resourceClass, ImageDescriptorBinding.ResourceClass(kind.Value)); + var described = ImageDescriptorBinding.Describe(kind.Value); + Assert.Equal(resourceClass, described.ResourceClass); + Assert.Equal(numericClass, described.NumericClass); + Assert.Equal(atomic, described.Atomic); Assert.Equal(BindingLayout.FirstImageBinding + index, BindingLayout.NativeBindingIndex(ShaderStage.Compute, kind.Value)); Assert.Equal((uint)DescriptorBindingKind.Count + BindingLayout.FirstImageBinding + index, BindingLayout.NativeBindingIndex(ShaderStage.Pixel, kind.Value)); index++; @@ -214,6 +220,10 @@ public sealed class BindingLayoutTests CheckBinding(ImageResourceClass.Sampled, ImageNumericClass.Float, ImageDimension.Dim2DArray, false, depthCompare: true); CheckBinding(ImageResourceClass.Sampled, ImageNumericClass.Float, ImageDimension.Dim2D, false, cube: true, depthCompare: true); + foreach (var dimension in storageDimensions) + CheckBinding(ImageResourceClass.Storage, ImageNumericClass.Sint, dimension, false); + CheckBinding(ImageResourceClass.Storage, ImageNumericClass.Sint, ImageDimension.Dim2D, false, cube: true); + Assert.Equal(BindingLayout.ImageBindingCount, index); static bool Invalid(ImageResource image) => ImageDescriptorBinding.ForImage(image) is null; @@ -237,7 +247,10 @@ public sealed class BindingLayoutTests invalid.ResourceClass = ImageResourceClass.Storage; invalid.Atomic = false; invalid.NumericClass = ImageNumericClass.Sint; + Assert.False(Invalid(invalid)); + invalid.Atomic = true; Assert.True(Invalid(invalid)); + invalid.Atomic = false; invalid.NumericClass = ImageNumericClass.Float; invalid.Dimension = ImageDimension.Dim2DMsaa; Assert.True(Invalid(invalid)); diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Resources/BufferBoundsCheckTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Resources/BufferBoundsCheckTests.cs new file mode 100644 index 0000000..03e7c03 --- /dev/null +++ b/tests/SharpEmu.ShaderCompiler.Tests/Resources/BufferBoundsCheckTests.cs @@ -0,0 +1,58 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.ShaderCompiler.Vulkan; +using Xunit; +using static SharpEmu.ShaderCompiler.Tests.Resources.ResourceTestProgram; + +namespace SharpEmu.ShaderCompiler.Tests.Resources; + +// A guest buffer word is bounds-checked in the shader unless the host measured that the device +// returns zero past the end of a descriptor range itself. +public sealed class BufferBoundsCheckTests +{ + [Theory] + [InlineData(false)] + [InlineData(true)] + public void BufferWordsAreCheckedUnlessTheDeviceZeroesOutOfRangeReads(bool deviceZeroes) + { + var program = Program(ScalarBufferLoad(0, 4, 20, count: 4), EndProgram(8)); + var (plan, resources, layout) = Prepare(program, SharpEmu.ShaderCompiler.Resources.ShaderStage.Compute, 0, 8); + var request = new ShaderCompileRequest(plan, resources, layout) + { + ZeroOutOfBoundsBufferReads = deviceZeroes, + }; + + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + + // One OpArrayLength is one range test. Without the device promise every word carries one. + var checks = CountOpcode(shader.Spirv, SpirvOp.ArrayLength); + if (deviceZeroes) + { + Assert.Equal(0, checks); + } + else + { + Assert.True(checks >= 4, $"expected a range test per word, found {checks}"); + } + } + + private static int CountOpcode(byte[] spirv, SpirvOp opcode) + { + var words = new uint[spirv.Length / 4]; + Buffer.BlockCopy(spirv, 0, words, 0, spirv.Length); + var count = 0; + for (var offset = 5; offset < words.Length;) + { + var wordCount = (int)(words[offset] >> 16); + if ((ushort)(words[offset] & 0xFFFF) == (ushort)opcode) + { + count++; + } + + offset += wordCount; + } + + return count; + } +} diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Resources/BufferCandidateTablePlannerTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Resources/BufferCandidateTablePlannerTests.cs index 800dec4..27028c0 100644 --- a/tests/SharpEmu.ShaderCompiler.Tests/Resources/BufferCandidateTablePlannerTests.cs +++ b/tests/SharpEmu.ShaderCompiler.Tests/Resources/BufferCandidateTablePlannerTests.cs @@ -22,7 +22,9 @@ public sealed class BufferCandidateTablePlannerTests string guard = "SCmpLtU32", uint guardRegister = 10, uint limit = 4, - bool runtimeLimit = false) + bool runtimeLimit = false, + uint descriptorOffset = 0, + bool secondField = false) { var instructions = new List(); uint pc = 0; @@ -41,8 +43,10 @@ public sealed class BufferCandidateTablePlannerTests var headerPc = pc; Add(Sop2(pc, "SMulI32", 12, Gen5Operand.Scalar(10), Operand(stride))); - Add(ScalarBufferLoad(pc, 0, destination: 16, count: 4, dynamicOffsetRegister: 12)); + Add(ScalarBufferLoad(pc, 0, destination: 16, count: secondField ? 8u : 4u, immediateOffset: (int)descriptorOffset, dynamicOffsetRegister: 12)); Add(BufferLoad(pc, 16, formatted: true)); + if (secondField) + Add(BufferLoad(pc, 20, formatted: true)); Add(Sop2(pc, "SAddU32", 10, Gen5Operand.Scalar(10), Operand(1))); var bound = runtimeLimit ? Gen5Operand.Scalar(7) : Operand(limit); Add(Sopc(pc, guard, Gen5Operand.Scalar(guardRegister), bound)); @@ -66,6 +70,55 @@ public sealed class BufferCandidateTablePlannerTests throw new Xunit.Sdk.XunitException("the program has no formatted buffer load"); } + [Fact] + public void TwoDescriptorFields_DoNotShareACandidateTable() + { + var plan = Extract(CandidateProgram(stride: 192, records: 4, descriptorOffset: 160, secondField: true)); + Assert.Equal(2, plan.BufferCandidateTables.Count); + Assert.Equal(new uint[] { 160, 176 }, plan.BufferCandidateTables.Select(table => table.BaseOffset)); + Assert.All(plan.BufferCandidateTables, table => Assert.Single(table.MemoryIndices)); + var memory = new TestWordMemory { Base = 0x3000, Words = new uint[0x1000 / 4], RequireAlignment = true }; + for (var record = 0; record < 4; record++) + for (var field = 0; field < 2; field++) + { + var address = 0x3000ul + (ulong)record * 192 + 160 + (ulong)field * 16; + memory.At(address) = 0x4000u + (uint)(record * 2 + field) * 0x100; + memory.At(address + 4) = 16u << 16; + memory.At(address + 8) = 4; + memory.At(address + 12) = 0; + } + var snapshot = new ResourceSnapshot(); + var specialization = new ResourceSpecialization(); + Assert.True(ResourceMaterializer.Materialize(plan, Inputs([], readCleanMemory: memory.Read), ref snapshot, ref specialization)); + Assert.Equal(2, specialization.BufferCandidateTables.Count); + for (var field = 0; field < 2; field++) + { + var table = specialization.BufferCandidateTables[field]; + Assert.Equal(4u, table.CandidateCount); + for (var record = 0; record < 4; record++) + Assert.Equal(0x4000u + (uint)(record * 2 + field) * 0x100, + snapshot.Buffers[(int)table.FirstCandidate + record][0]); + } + } + + [Fact] + public void DescriptorOffsetBeyondTheAddressSpace_IsRejected() + { + var plan = Extract(CandidateProgram(stride: 192, records: 4, descriptorOffset: uint.MaxValue - 15)); + Assert.Empty(plan.BufferCandidateTables); + } + + [Fact] + public void DescriptorInsideAStridedRecord_KeepsItsFieldOffset() + { + var plan = Extract(CandidateProgram(stride: 192, records: 4, descriptorOffset: 160)); + var table = Assert.Single(plan.BufferCandidateTables); + Assert.Equal(160u, table.BaseOffset); + Assert.Equal(160u, table.MinOffset); + Assert.Equal(752u, table.MaxOffset); + Assert.Equal(352u, table.StaticCandidateOffset(1)); + } + [Fact] public void CanonicalUnsignedLoop_ProducesCandidateTable() { diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Resources/BufferFormatLoadTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Resources/BufferFormatLoadTests.cs new file mode 100644 index 0000000..51d2cda --- /dev/null +++ b/tests/SharpEmu.ShaderCompiler.Tests/Resources/BufferFormatLoadTests.cs @@ -0,0 +1,61 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Diagnostics; +using SharpEmu.ShaderCompiler.Vulkan; +using Xunit; +using static SharpEmu.ShaderCompiler.Tests.Resources.ResourceTestProgram; + +namespace SharpEmu.ShaderCompiler.Tests.Resources; + +// A formatted untyped load decodes the descriptor format at run time. Dword-aligned 32-bit +// raw-bit formats take a word-per-component branch; every other format the byte-wise decode. +public sealed class BufferFormatLoadTests +{ + [Theory] + [InlineData(1u)] + [InlineData(3u)] + [InlineData(4u)] + public void FormattedLoadBranchesBetweenTheWordAndByteDecodes(uint dwords) + { + // Only the run-time format decoder is under test, not the decode a known format selects. + var (plan, resources, layout) = Prepare(Program(BufferLoad(0, 4, dwords: dwords, formatted: true), EndProgram(8))); + var request = new ShaderCompileRequest(plan, resources, layout) { ForceGenericBufferFormats = true }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + + var module = new SpirvModuleInspector(shader.Spirv); + // The word branch and the byte-wise branch join at one selection merge. + Assert.Contains((ushort)SpirvOp.SelectionMerge, module.Opcodes); + Assert.Contains((ushort)SpirvOp.BitFieldUExtract, module.Opcodes); + ValidateWhenAvailable(shader.Spirv); + } + + private static void ValidateWhenAvailable(byte[] code) + { + var sdk = Environment.GetEnvironmentVariable("VULKAN_SDK"); + var executable = string.IsNullOrWhiteSpace(sdk) ? null + : Path.Combine(sdk, OperatingSystem.IsWindows() ? "Bin/spirv-val.exe" : "bin/spirv-val"); + if (executable is null || !File.Exists(executable)) + { + return; + } + + var path = Path.GetTempFileName(); + try + { + File.WriteAllBytes(path, code); + var start = new ProcessStartInfo(executable) { UseShellExecute = false, RedirectStandardError = true, CreateNoWindow = true }; + start.ArgumentList.Add("--target-env"); + start.ArgumentList.Add("vulkan1.2"); + start.ArgumentList.Add(path); + using var process = Process.Start(start)!; + var error = process.StandardError.ReadToEnd(); + process.WaitForExit(); + Assert.True(process.ExitCode == 0, error); + } + finally + { + File.Delete(path); + } + } +} diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Resources/BufferUnalignedWordTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Resources/BufferUnalignedWordTests.cs new file mode 100644 index 0000000..03cb13b --- /dev/null +++ b/tests/SharpEmu.ShaderCompiler.Tests/Resources/BufferUnalignedWordTests.cs @@ -0,0 +1,46 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.ShaderCompiler.Vulkan; +using Xunit; +using static SharpEmu.ShaderCompiler.Tests.Resources.ResourceTestProgram; + +namespace SharpEmu.ShaderCompiler.Tests.Resources; + +// The byte-wise decode of a formatted buffer load reads the four bytes of a component out of +// the two words they span, not out of four separately bounds-checked words. +public sealed class BufferUnalignedWordTests +{ + [Fact] + public void ByteWiseComponentDecodeReadsTwoBoundsCheckedWords() + { + // Only the run-time format decoder is under test, not the decode a known format selects. + var (plan, resources, layout) = Prepare(Program(BufferLoad(0, 4, dwords: 1, formatted: true), EndProgram(8))); + var request = new ShaderCompileRequest(plan, resources, layout) { ForceGenericBufferFormats = true }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + + // One bounds-checked buffer word load is one OpArrayLength. A formatted load decodes + // four components whatever the opcode's width; the byte-wise branch now reads two words + // per component instead of four, which is eight of these instead of sixteen. + Assert.Equal(24, CountOpcode(shader.Spirv, SpirvOp.ArrayLength)); + } + + private static int CountOpcode(byte[] spirv, SpirvOp opcode) + { + var words = new uint[spirv.Length / 4]; + Buffer.BlockCopy(spirv, 0, words, 0, spirv.Length); + var count = 0; + for (var offset = 5; offset < words.Length;) + { + var wordCount = (int)(words[offset] >> 16); + if ((ushort)(words[offset] & 0xFFFF) == (ushort)opcode) + { + count++; + } + + offset += wordCount; + } + + return count; + } +} diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Resources/DescriptorReadTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Resources/DescriptorReadTests.cs new file mode 100644 index 0000000..b31772a --- /dev/null +++ b/tests/SharpEmu.ShaderCompiler.Tests/Resources/DescriptorReadTests.cs @@ -0,0 +1,75 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using System.Reflection; +using SharpEmu.ShaderCompiler.Resources; +using Xunit; + +namespace SharpEmu.ShaderCompiler.Tests.Resources; + +public sealed class DescriptorReadTests +{ + private delegate bool RangeRead(ReadOnlySpan descriptor, uint dynamicOffset, + uint immediateOffset, ResourceRuntimeInputs inputs, Span words); + + private static RangeRead Reader => typeof(ResourceMaterializer).Assembly + .GetType("SharpEmu.ShaderCompiler.Resources.ScalarBufferRangeRead")! + .GetMethod("TryRead", BindingFlags.Static | BindingFlags.NonPublic)!.CreateDelegate(); + + [Fact] + public void ContiguousDescriptor_UsesOneReadAndPreservesWords() + { + var calls = 0; + var inputs = new ResourceRuntimeInputs + { + ReadCleanWords = (ulong address, Span words) => + { + Assert.Equal(0x1020UL, address); + calls++; + for (var i = 0; i < words.Length; i++) words[i] = (uint)(100 + i); + return true; + }, + }; + var result = new uint[8]; + Assert.True(Reader([0x1000, 0, 64, 0], 32, 0, inputs, result)); + Assert.Equal(1, calls); + Assert.Equal(Enumerable.Range(100, 8).Select(i => (uint)i), result); + } + + [Theory] + [InlineData(48u, 0u)] // Partial OOB must retain individual zero-filling semantics. + [InlineData(0u, uint.MaxValue)] // Immediate offsets must not wrap. + public void OutOfBoundsRange_DoesNotReadGuestMemory(uint offset, uint immediate) + { + var calls = 0; + var inputs = new ResourceRuntimeInputs { ReadCleanWords = (ulong _, Span _) => { calls++; return true; } }; + Assert.False(Reader([0x1000, 0, 64, 0], offset, immediate, inputs, new uint[8])); + Assert.Equal(0, calls); + } + + [Fact] + public void GpuOwnedRange_RefusesBulkRead() + { + var inputs = new ResourceRuntimeInputs { ReadCleanWords = (ulong _, Span _) => false }; + Assert.False(Reader([0x1000, 0, 64, 0], 0, 0, inputs, new uint[8])); + } + + [Fact] + public void ChangingObservation_IsUncacheableAndRepeatedReadsAreDeduplicated() + { + var type = typeof(ResourceMaterializationCache).GetNestedType("ReadRecorder", BindingFlags.NonPublic)!; + var recorder = Activator.CreateInstance(type, nonPublic: true)!; + uint value = 1; + GuestWordReader source = (ulong _, out uint word) => { word = value; return true; }; + var reader = (GuestWordReader)type.GetMethod("Wrap", [typeof(GuestWordReader), typeof(bool)])! + .Invoke(recorder, [source, true])!; + Assert.True(reader(0x1000, out _)); + Assert.True(reader(0x1000, out _)); + var reads = (System.Collections.ICollection)type.GetProperty("Reads")!.GetValue(recorder)!; + Assert.Single(reads.Cast()); + Assert.False((bool)type.GetProperty("Failed")!.GetValue(recorder)!); + value = 2; + Assert.True(reader(0x1000, out _)); + Assert.True((bool)type.GetProperty("Failed")!.GetValue(recorder)!); + } +} diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Resources/DirectImageTableTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Resources/DirectImageTableTests.cs index f5ba2bb..1cdcd2d 100644 --- a/tests/SharpEmu.ShaderCompiler.Tests/Resources/DirectImageTableTests.cs +++ b/tests/SharpEmu.ShaderCompiler.Tests/Resources/DirectImageTableTests.cs @@ -11,6 +11,24 @@ namespace SharpEmu.ShaderCompiler.Tests.Resources; public sealed class DirectImageTableTests { + [Theory] + [InlineData(0x80000000u, true)] + [InlineData(0x10000000u, false)] + public void ImageCandidatesDistinguishResourceLevelFromReservedBits(uint bit, bool valid) + { + var plan = ShaderResourcePlan.Extract(CreateWaveIndexedDescriptorProgram(), ShaderStage.Compute, Hash, 0, 2); + bool Read(ulong address, out uint word) + { + if (!ReadWaveIndexedMemory(address, out word)) return false; + if (address >= 0x1100 && (address - 0x1100) % 32 == 8) word |= bit; + return true; + } + ResourceSnapshot snapshot = new(); ResourceSpecialization specialization = new(); + Assert.True(ResourceMaterializer.Materialize(plan, Inputs([0x1000, 0], readCleanMemory: Read), ref snapshot, ref specialization)); + Assert.Equal(valid, snapshot.Images.Any(image => image[0] != 0)); + if (valid) Assert.All(snapshot.Images, image => Assert.Equal(bit, image[2] & bit)); + } + internal static Gen5ShaderProgram CreateWaveIndexedDescriptorProgram() { return Program( @@ -166,6 +184,29 @@ public sealed class DirectImageTableTests Assert.Single(snapshot.Images, image => image.All(word => word == 0)); } + [Fact] + public void IndirectTableWithOnlyIncompatibleDimensionsBindsTypedNullCandidates() + { + var program = CreateWaveIndexedDescriptorProgram(); + program = program with + { + Instructions = program.Instructions.Select(instruction => + instruction.Pc == 64 + ? Image(64, "ImageLoad", 4, dimension: 6, dmask: 1, vectorAddress: 1) + : instruction).ToArray(), + }; + var plan = ShaderResourcePlan.Extract(program, ShaderStage.Compute, Hash, 0, 2); + var snapshot = new ResourceSnapshot(); + var specialization = new ResourceSpecialization(); + + Assert.True(ResourceMaterializer.Materialize( + plan, Inputs([0x1000, 0], readCleanMemory: ReadWaveIndexedMemory), + ref snapshot, ref specialization, out var failure), $"materialize {failure}"); + Assert.Equal(2, snapshot.Images.Length); + Assert.All(snapshot.Images, image => Assert.All(image, word => Assert.Equal(0u, word))); + Assert.All(specialization.Images, image => Assert.Equal(ImageDimension.Dim2DMsaa, image.Dimension)); + } + [Theory] [InlineData(false)] [InlineData(true)] @@ -190,11 +231,16 @@ public sealed class DirectImageTableTests Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out _, out var error), error); } + // Without the restored EXEC the loop is no waterfall: no wave-indexed table is planned, + // and the load reads the descriptor its registers hold at run time. [Fact] public void ReadLaneWithoutRestoredExecutionIsNotWaveIndexed() { - Assert.Throws(() => - ShaderResourcePlan.Extract(CreateWaveIndexedReadLaneProgram(selfAddressed: true, restoreExec: false), ShaderStage.Compute, Hash, 0, 2)); + var plan = ShaderResourcePlan.Extract(CreateWaveIndexedReadLaneProgram(selfAddressed: true, restoreExec: false), ShaderStage.Compute, Hash, 0, 2); + + Assert.Empty(plan.IndirectImages); + Assert.Contains(plan.Memory.Entries, entry => entry.RuntimeDescriptor); + Assert.Empty(plan.Info.NullDescriptorFallbacks); } private static bool ReadWaveIndexedMemory(ulong address, out uint word) @@ -340,10 +386,15 @@ public sealed class DirectImageTableTests } } + // An unbounded index still plans no direct table; the load reads its descriptor at run time. [Fact] - public void UnboundedDirectTableIsStillRejected() + public void UnboundedDirectTableIsReadAtRuntime() { - Assert.Throws(() => ShaderResourcePlan.Extract(CreateProgram(bitScan: false), ShaderStage.Compute, Hash, 0, 2)); + var plan = ShaderResourcePlan.Extract(CreateProgram(bitScan: false), ShaderStage.Compute, Hash, 0, 2); + + Assert.Empty(plan.IndirectImages); + Assert.Contains(plan.Memory.Entries, entry => entry.RuntimeDescriptor); + Assert.Empty(plan.Info.NullDescriptorFallbacks); } [Theory] @@ -552,11 +603,11 @@ public sealed class DirectImageTableTests bool Read(ulong address, out uint word) { var success = ReadDescriptor(address, out word); - if (address == 0x1000 + 344 + 12) - { - if (!incompatible) return false; - word = (word & 0x0FFFFFFF) | (11u << 28); - } + // A converted format cannot share a case with a directly sampled one. + if (incompatible && address == 0x1000 + 344 + 4) + word = (word & ~(0x1FFu << 20)) | (GuestImageFormat.Format11x2x10Uint << 20); + if (!incompatible && address == 0x1000 + 344 + 12) + return false; return success; } Assert.False(ResourceMaterializer.Materialize(plan, Inputs([0x1000, 0], readCleanMemory: Read), ref snapshot, ref specialization, @@ -565,4 +616,50 @@ public sealed class DirectImageTableTests Assert.Same(original, snapshot); Assert.Same(originalSpecialization, specialization); } + + // Vulkan cannot sample a multisample image. A sample instruction reads sample 0 of the texel + // its coordinate falls in (as shadPS4 lowers it); a gather has no such reading and is refused. + [Theory] + [InlineData("ImageSampleA", true)] + [InlineData("ImageSampleLz", true)] + [InlineData("ImageGather4Lz", false)] + [InlineData("ImageLoad", true)] + public void MultisampleDescriptorsRequireLoadsInsteadOfInvalidSamplingModules(string opcode, bool accepted) + { + var gather = opcode.StartsWith("ImageGather", StringComparison.Ordinal); + var program = Program([Image(0, opcode, 0, samplerRegister: 8, dmask: 1), EndProgram(8)]); + var plan = ShaderResourcePlan.Extract(program, ShaderStage.Pixel, Hash, 0, 12); + var inputs = new ResourceRuntimeInputs { UserData = [0x1000, GuestImageFormat.Format32Float << 20, 0, + 0xFAC | (1u << 16) | (GuestImageFormat.ImageType2DMsaa << 28), 0, 0x10, 0, 0, + 0, 0xFFF000, 0, 0] }; + ResourceSnapshot snapshot = new(); ResourceSpecialization specialization = new(); + Assert.True(ResourceMaterializer.Materialize(plan, inputs, ref snapshot, ref specialization)); + var resources = ResourceMaterializer.ApplyTo(plan, specialization); + Assert.Equal(ImageDimension.Dim2DMsaa, resources.Info.Images[0].Dimension); + var layout = BindingLayout.Allocate(resources.Info, BindingLayout.CollectUserDataRegisters(program, 0, 12), + false, ShaderCompileRequest.RequiresFlattenedTable(plan, resources), false); + Assert.Equal(accepted, Gen5SpirvTranslator.TryCompileProgram(new ShaderCompileRequest(plan, resources, layout), out var shader, out var error)); + if (gather) Assert.Contains("multisample sampling semantics", error); + if (accepted) Gen5LargeDispatcherValidationTests.ValidateWithSpirvToolsWhenAvailable(shader.Spirv); + } + + // A table in a scalar buffer, indexed by a lane value: planned as a dense table it binds + // every record of the buffer on each draw. An access the runtime table can read binds only + // the records it reads; one it cannot (a storage write) keeps the dense table. + [Theory] + [InlineData("ImageLoad", true)] + [InlineData("ImageStore", false)] + public void BufferTableImagesAreReadAtRuntimeWhenTheAccessAllowsIt(string opcode, bool runtime) + { + var program = Program( + ReadFirstLane(0, 106, 0), + Sop2(4, "SMulI32", 106, Gen5Operand.Scalar(106), Operand(384)), + ScalarBufferLoad(12, 0, 16, 8, dynamicOffsetRegister: 106), + Image(20, opcode, 16, dmask: 1, vectorAddress: 4), + EndProgram(28)); + var plan = Extract(program, userDataCount: 4); + + Assert.Equal(runtime, plan.Info.UsesRuntimeDescriptors); + Assert.Equal(!runtime, plan.DescriptorSources.Any(source => source.IndirectImage?.BufferTableStride == 384)); + } } diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Resources/IndirectSelectorValuesTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Resources/IndirectSelectorValuesTests.cs index 33a5ff5..38abe30 100644 --- a/tests/SharpEmu.ShaderCompiler.Tests/Resources/IndirectSelectorValuesTests.cs +++ b/tests/SharpEmu.ShaderCompiler.Tests/Resources/IndirectSelectorValuesTests.cs @@ -127,7 +127,8 @@ public sealed class IndirectSelectorValuesTests memory.At(0x1000 + 36) = 1; var first = ResourceTrackerTests.ImageDescriptor(); var second = first.ToArray(); - second[3] = (second[3] & 0x0FFFFFFF) | (10u << 28); + // A converted format cannot share a case with a directly sampled one. + second[1] = (second[1] & ~(0x1FFu << 20)) | (GuestImageFormat.Format11x2x10Uint << 20); ResourceTrackerTests.WriteImage(memory, 0x2000, first); ResourceTrackerTests.WriteImage(memory, 0x2020, second); var snapshot = new ResourceSnapshot(); @@ -167,7 +168,8 @@ public sealed class IndirectSelectorValuesTests memory.At(0x1000 + 36) = 1; var first = ResourceTrackerTests.ImageDescriptor(); var second = first.ToArray(); - second[3] = (second[3] & 0x0FFFFFFF) | (10u << 28); + // A converted format cannot share a case with a directly sampled one. + second[1] = (second[1] & ~(0x1FFu << 20)) | (GuestImageFormat.Format11x2x10Uint << 20); ResourceTrackerTests.WriteImage(memory, 0x2000, first); ResourceTrackerTests.WriteImage(memory, 0x2020, second); var captures = new List(); diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Resources/PlannedScalarTableLoadTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Resources/PlannedScalarTableLoadTests.cs new file mode 100644 index 0000000..59edbbc --- /dev/null +++ b/tests/SharpEmu.ShaderCompiler.Tests/Resources/PlannedScalarTableLoadTests.cs @@ -0,0 +1,87 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.ShaderCompiler; +using SharpEmu.ShaderCompiler.Resources; +using SharpEmu.ShaderCompiler.Vulkan; +using Xunit; +using static SharpEmu.ShaderCompiler.Tests.Resources.ResourceTestProgram; + +namespace SharpEmu.ShaderCompiler.Tests.Resources; + +public sealed class PlannedScalarTableLoadTests +{ + [Theory] + [InlineData(false, 0)] + [InlineData(false, 1)] + [InlineData(false, 2)] + [InlineData(true, 0)] + [InlineData(true, 1)] + [InlineData(true, 2)] + public void ScalarLoadsReadGuestWordsAfterTheImageSlotPrefix(bool bindless, int imageCount) + { + var instructions = new List(); + uint pc = 0; + for (var image = 0; image < imageCount; image++) + { + var register = (uint)(32 + image * 8); + uint[] words = [(uint)(0x1000 + image * 0x1000), 22u << 20, 3u | (3u << 14), 0xFACu | (9u << 28), 0, 0, 0, 0]; + for (uint component = 0; component < words.Length; component++) + { + instructions.Add(MoveScalar(pc, register + component, words[component])); + pc += 8; + } + instructions.Add(Image(pc, "ImageStore", register)); + pc += 8; + } + var scalarPc = pc; + instructions.Add(ScalarLoad(pc, 0, destination: 16, count: 4)); pc += 8; + for (uint component = 0; component < 4; component++) + { + instructions.Add(MoveVectorFromScalar(pc, 4, 16 + component)); pc += 4; + instructions.Add(BufferStore(pc, 4, (int)(component * 4))); pc += 8; + } + instructions.Add(EndProgram(pc)); + var request = Request(Program([.. instructions]), userDataCount: 9, usesBindlessImages: bindless); + Assert.Equal((uint)imageCount, BindingLayout.ImageSlotTableDwordCount(request.Resources.Info)); + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + var names = new SpirvModuleInspector(shader.Spirv).Names; + var tableId = Assert.Single(names, pair => pair.Value == "flattenedResourceTable").Key; + var constants = new Dictionary(); + var pointers = new Dictionary(); + var loads = new Dictionary(); + var loadedSlots = new Dictionary(); + var wordsSpirv = new uint[shader.Spirv.Length / 4]; + Buffer.BlockCopy(shader.Spirv, 0, wordsSpirv, 0, shader.Spirv.Length); + for (var offset = 5; offset < wordsSpirv.Length;) + { + var count = (int)(wordsSpirv[offset] >> 16); + var operands = wordsSpirv.AsSpan(offset + 1, count - 1); + switch ((SpirvOp)(wordsSpirv[offset] & 0xFFFF)) + { + case SpirvOp.Constant when count == 4: + constants[operands[1]] = operands[2]; + break; + case SpirvOp.AccessChain when operands[2] == tableId && constants.TryGetValue(operands[^1], out var constantSlot): + pointers[operands[1]] = constantSlot; + break; + case SpirvOp.Load when pointers.TryGetValue(operands[2], out var slot): + loads[operands[1]] = slot; + break; + case SpirvOp.Store when loads.TryGetValue(operands[1], out var loaded): + loadedSlots[operands[0]] = loaded; + break; + } + offset += count; + } + for (uint component = 0; component < 4; component++) + { + Assert.True(request.Memory.TryGetIndex(scalarPc, component, out var memoryIndex)); + var flatSlot = request.FlattenedSlotByMemoryIndex[memoryIndex]; + Assert.True(flatSlot < request.FlattenedTableReservedWords); + var register = Assert.Single(names, pair => pair.Value == $"s{16 + component}").Key; + // The host binds image indices first, then the materialized guest words. + Assert.Equal(flatSlot + (bindless ? (uint)imageCount : 0), loadedSlots[register]); + } + } +} diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Resources/ResourceMaterializationCacheTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Resources/ResourceMaterializationCacheTests.cs index e1f7354..be8e032 100644 --- a/tests/SharpEmu.ShaderCompiler.Tests/Resources/ResourceMaterializationCacheTests.cs +++ b/tests/SharpEmu.ShaderCompiler.Tests/Resources/ResourceMaterializationCacheTests.cs @@ -1,279 +1,348 @@ -// Copyright (C) 2026 SharpEmu Emulator Project -// SPDX-License-Identifier: GPL-2.0-or-later - -using SharpEmu.ShaderCompiler.Resources; -using Xunit; -using static SharpEmu.ShaderCompiler.Tests.Resources.ResourceTestProgram; - -namespace SharpEmu.ShaderCompiler.Tests.Resources; - -public sealed class ResourceMaterializationCacheTests -{ - private const ulong HeapBase = 0x1000; - - // A mask word, a two-entry index table and six 32-byte image records. - private sealed class Heap - { - public readonly Dictionary Words = new(); - public int Reads; - public bool GpuOwned; - - public Heap() - { - Words[HeapBase + 0x80] = (1u << 1) | (1u << 4); - Words[HeapBase + 0x40 + 0x90] = 2; - Words[HeapBase + 0x40 + 4 * 0x90] = 5; - foreach (var record in new uint[] { 2, 5 }) - { - var entry = HeapBase + 0x100 + record * 32; - for (uint dword = 0; dword < 8; dword++) - Words[entry + dword * 4] = dword switch - { - 0 => (record & 1) == 0 ? 0x2000u : 0x1000u, - 1 => 20u << 20, - 3 => 0xFACu | (9u << 28), - _ => 0, - }; - } - } - - public bool Read(ulong address, out uint word) - { - Reads++; - return Words.TryGetValue(address, out word); - } - - public bool ReadResident(ulong address, Span destination, bool clean) - { - if (GpuOwned) - return false; - for (var offset = 0; offset < destination.Length; offset += 4) - { - if (!Words.TryGetValue(address + (ulong)offset, out var word)) - return false; - BitConverter.TryWriteBytes(destination[offset..], word); - } - - return true; - } - } - - private static ShaderResourcePlan Plan() => - ShaderResourcePlan.Extract(DirectImageTableTests.CreateWaveIndexedDescriptorProgram(), ShaderStage.Compute, Hash, 0, 2); - - private static bool Run(ResourceMaterializationCache cache, ShaderResourcePlan plan, Heap heap, uint[] userData, - out ResourceSnapshot snapshot, out ResourceSpecialization specialization, ulong shaderBase = 0) - { - snapshot = new ResourceSnapshot(); - specialization = new ResourceSpecialization(); - return cache.Materialize(plan, Inputs(userData, readCleanMemory: heap.Read, shaderBase: shaderBase), heap.ReadResident, - ref snapshot, ref specialization, out _); - } - - [Fact] - public void UnchangedMemoryReusesTheMaterialization() - { - var plan = Plan(); - var heap = new Heap(); - var cache = new ResourceMaterializationCache(); - Assert.True(Run(cache, plan, heap, [0x1000, 0], out var first, out var firstSpecialization)); - var readsAfterFirst = heap.Reads; - Assert.True(readsAfterFirst > 0); - - Assert.True(Run(cache, plan, heap, [0x1000, 0], out var second, out var secondSpecialization)); - Assert.Equal(readsAfterFirst, heap.Reads); - Assert.Same(first, second); - Assert.Same(firstSpecialization, secondSpecialization); - Assert.Equal((1, 1), (cache.Hits, cache.Misses)); - } - - [Fact] - public void AChangedDescriptorWordMaterializesAgain() - { - var plan = Plan(); - var heap = new Heap(); - var cache = new ResourceMaterializationCache(); - Assert.True(Run(cache, plan, heap, [0x1000, 0], out var first, out _)); - - heap.Words[HeapBase + 0x100 + 5 * 32] = 0x3000; - Assert.True(Run(cache, plan, heap, [0x1000, 0], out var second, out _)); - Assert.NotSame(first, second); - Assert.Equal((0, 2), (cache.Hits, cache.Misses)); - - // An uncached full walk gives the same result as the re-materialization. - var reference = new ResourceSnapshot(); - var referenceSpecialization = new ResourceSpecialization(); - Assert.True(ResourceMaterializer.Materialize(plan, Inputs([0x1000, 0], readCleanMemory: heap.Read), - ref reference, ref referenceSpecialization)); - Assert.Equal(reference.Images.Select(image => image.ToArray()), second.Images.Select(image => image.ToArray())); - } - - [Fact] - public void AlternatingDescriptorsReuseEveryRecentVariant() - { - var plan = Plan(); - var heap = new Heap(); - var cache = new ResourceMaterializationCache(); - var word = HeapBase + 0x100 + 5 * 32; - Assert.True(Run(cache, plan, heap, [0x1000, 0], out var first, out _)); - heap.Words[word] = 0x3000; - Assert.True(Run(cache, plan, heap, [0x1000, 0], out var second, out _)); - Assert.Equal((0, 2), (cache.Hits, cache.Misses)); - - heap.Words[word] = 0x1000; - Assert.True(Run(cache, plan, heap, [0x1000, 0], out var firstAgain, out _)); - heap.Words[word] = 0x3000; - Assert.True(Run(cache, plan, heap, [0x1000, 0], out var secondAgain, out _)); - Assert.Same(first, firstAgain); - Assert.Same(second, secondAgain); - Assert.Equal((2, 2), (cache.Hits, cache.Misses)); - } - - [Fact] - public void AChangedMaskMaterializesAgain() - { - var plan = Plan(); - var heap = new Heap(); - var cache = new ResourceMaterializationCache(); - Assert.True(Run(cache, plan, heap, [0x1000, 0], out _, out _)); - heap.Words[HeapBase + 0x80] = 1u << 1; - Assert.True(Run(cache, plan, heap, [0x1000, 0], out _, out _)); - Assert.Equal(0, cache.Hits); - } - - [Fact] - public void GpuOwnedMemoryIsNeverTrusted() - { - var plan = Plan(); - var heap = new Heap(); - var cache = new ResourceMaterializationCache(); - Assert.True(Run(cache, plan, heap, [0x1000, 0], out _, out _)); - heap.GpuOwned = true; - Assert.True(Run(cache, plan, heap, [0x1000, 0], out _, out _)); - Assert.Equal(0, cache.Hits); - } - - [Fact] - public void ADifferentDrawKeyIsADifferentEntry() - { - var plan = Plan(); - var heap = new Heap(); - var cache = new ResourceMaterializationCache(); - Assert.True(Run(cache, plan, heap, [0x1000, 0], out var first, out _)); - Assert.True(Run(cache, plan, heap, [0x1000, 0], out var second, out _, shaderBase: 0x100)); - Assert.NotSame(first, second); - Assert.True(Run(cache, plan, heap, [0x1000, 0], out var third, out _)); - Assert.Same(first, third); - Assert.Equal((1, 2), (cache.Hits, cache.Misses)); - } - - [Fact] - public void AFailedReadIsNotCached() - { - var plan = Plan(); - var heap = new Heap(); - heap.Words.Remove(HeapBase + 0x100 + 5 * 32 + 4); - var cache = new ResourceMaterializationCache(); - var ok = Run(cache, plan, heap, [0x1000, 0], out _, out _); - var reads = heap.Reads; - Assert.Equal(ok, Run(cache, plan, heap, [0x1000, 0], out _, out _)); - Assert.True(heap.Reads > reads); - Assert.Equal(0, cache.Hits); - } - - [Fact] - public void AFailedRecordingDoesNotPoisonTheNextReader() - { - var plan = Plan(); - var incomplete = new Heap(); - incomplete.Words.Remove(HeapBase + 0x100 + 5 * 32 + 4); - var cache = new ResourceMaterializationCache(); - Run(cache, plan, incomplete, [0x1000, 0], out _, out _); - - var complete = new Heap(); - Assert.True(Run(cache, plan, complete, [0x1000, 0], out var first, out _)); - Assert.True(Run(cache, plan, complete, [0x1000, 0], out var second, out _)); - Assert.Same(first, second); - Assert.Equal(1, cache.Hits); - - // Retained entries must not refer to scratch reads reused for another key. - complete.Words[HeapBase + 0x80] = 1u << 1; - Assert.True(Run(cache, plan, complete, [0x1000, 0], out _, out _, shaderBase: 0x100)); - var original = new Heap(); - Assert.True(Run(cache, plan, original, [0x1000, 0], out var restored, out _)); - Assert.Same(first, restored); - } - - // The table pointer in s[0:1]; the rest of the nine user-data registers the program declares. - private static readonly uint[] TableUserData = [0x1000, 0, 0, 0, 0, 0, 0, 0, 0]; - - // Reads the four constants of FlattenedReadReuseTests.RepeatedReadProgram from a word memory. - private static bool RunTable(ResourceMaterializationCache cache, ShaderResourcePlan plan, TestWordMemory memory, - out ResourceSnapshot snapshot) - { - snapshot = new ResourceSnapshot(); - var specialization = new ResourceSpecialization(); - return cache.Materialize(plan, Inputs(TableUserData, memory.Read, memory.Read), (address, destination, _) => - { - for (var offset = 0; offset < destination.Length; offset += 4) - { - if (!memory.Read(address + (ulong)offset, out var word)) - return false; - BitConverter.TryWriteBytes(destination[offset..], word); - } - - return true; - }, ref snapshot, ref specialization, out _); - } - - [Fact] - public void AChangedTableOnlyWordRefreshesOnlyTheTable() - { - var (plan, _, _) = Prepare(FlattenedReadReuseTests.RepeatedReadProgram(4), userDataCount: 9); - var memory = new TestWordMemory { Words = [11, 22, 33, 44] }; - var cache = new ResourceMaterializationCache(); - Assert.True(RunTable(cache, plan, memory, out var first)); - Assert.Equal([11u, 22, 33, 44], first.FlattenedResourceTable); - - memory.Words[2] = 99; - Assert.True(RunTable(cache, plan, memory, out var second)); - Assert.Equal((0, 1, 1), (cache.Hits, cache.Misses, cache.TableRefreshes)); - Assert.Equal([11u, 22, 99, 44], second.FlattenedResourceTable); - Assert.Same(first.Buffers, second.Buffers); - - // The refreshed table matches an uncached full walk, and the refreshed entry is then a hit. - var reference = new ResourceSnapshot(); - var referenceSpecialization = new ResourceSpecialization(); - Assert.True(ResourceMaterializer.Materialize(plan, Inputs(TableUserData, memory.Read, memory.Read), ref reference, ref referenceSpecialization)); - Assert.Equal(reference.FlattenedResourceTable, second.FlattenedResourceTable); - Assert.True(RunTable(cache, plan, memory, out var third)); - Assert.Same(second, third); - Assert.Equal((1, 1, 1), (cache.Hits, cache.Misses, cache.TableRefreshes)); - } - - [Fact] - public void AnUnreadableTableWordFallsBackToAFullWalk() - { - var (plan, _, _) = Prepare(FlattenedReadReuseTests.RepeatedReadProgram(4), userDataCount: 9); - var memory = new TestWordMemory { Words = [11, 22, 33, 44] }; - var cache = new ResourceMaterializationCache(); - Assert.True(RunTable(cache, plan, memory, out _)); - memory.Words[1] = 7; - memory.FailAddress = 0x1000 + 3 * 4; - RunTable(cache, plan, memory, out _); - Assert.Equal(0, cache.TableRefreshes); - } - - [Fact] - public void AFullGenerationKeepsRecentEntries() - { - var plan = Plan(); - var heap = new Heap(); - var cache = new ResourceMaterializationCache(generationCapacity: 1); - Assert.True(Run(cache, plan, heap, [0x1000, 0], out _, out _)); - Assert.True(Run(cache, plan, heap, [0x1000, 0], out _, out _, shaderBase: 0x100)); - // The first entry moved to the old generation and is still found. - Assert.True(Run(cache, plan, heap, [0x1000, 0], out _, out _)); - Assert.Equal(1, cache.Hits); - } -} +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.ShaderCompiler.Resources; +using Xunit; +using static SharpEmu.ShaderCompiler.Tests.Resources.ResourceTestProgram; + +namespace SharpEmu.ShaderCompiler.Tests.Resources; + +public sealed class ResourceMaterializationCacheTests +{ + private const ulong HeapBase = 0x1000; + + // A mask word, a two-entry index table and six 32-byte image records. + private sealed class Heap + { + public readonly Dictionary Words = new(); + public int Reads; + public bool GpuOwned; + + public Heap() + { + Words[HeapBase + 0x80] = (1u << 1) | (1u << 4); + Words[HeapBase + 0x40 + 0x90] = 2; + Words[HeapBase + 0x40 + 4 * 0x90] = 5; + foreach (var record in new uint[] { 2, 5 }) + { + var entry = HeapBase + 0x100 + record * 32; + for (uint dword = 0; dword < 8; dword++) + Words[entry + dword * 4] = dword switch + { + 0 => (record & 1) == 0 ? 0x2000u : 0x1000u, + 1 => 20u << 20, + 3 => 0xFACu | (9u << 28), + _ => 0, + }; + } + } + + public bool Read(ulong address, out uint word) + { + Reads++; + return Words.TryGetValue(address, out word); + } + + public bool ReadResident(ulong address, Span destination, bool clean) + { + if (GpuOwned) + return false; + for (var offset = 0; offset < destination.Length; offset += 4) + { + if (!Words.TryGetValue(address + (ulong)offset, out var word)) + return false; + BitConverter.TryWriteBytes(destination[offset..], word); + } + + return true; + } + } + + private static ShaderResourcePlan Plan() => + ShaderResourcePlan.Extract(DirectImageTableTests.CreateWaveIndexedDescriptorProgram(), ShaderStage.Compute, Hash, 0, 2); + + private static bool Run(ResourceMaterializationCache cache, ShaderResourcePlan plan, Heap heap, uint[] userData, + out ResourceSnapshot snapshot, out ResourceSpecialization specialization, ulong shaderBase = 0) + { + snapshot = new ResourceSnapshot(); + specialization = new ResourceSpecialization(); + return cache.Materialize(plan, Inputs(userData, readCleanMemory: heap.Read, shaderBase: shaderBase), heap.ReadResident, + ref snapshot, ref specialization, out _); + } + + [Fact] + public void GroupedDescriptorReads_PreserveMaterializationAndCacheInvalidation() + { + var plan = Plan(); + var heap = new Heap(); + var cache = new ResourceMaterializationCache(); + var bulkReads = 0; + var inputs = Inputs([0x1000, 0], readCleanMemory: heap.Read) with + { + ReadCleanWords = (ulong address, Span words) => + { + bulkReads++; + for (var i = 0; i < words.Length; i++) + if (!heap.Words.TryGetValue(address + (ulong)i * 4, out words[i])) return false; + return !heap.GpuOwned; + }, + }; + var snapshot = new ResourceSnapshot(); + var specialization = new ResourceSpecialization(); + Assert.True(cache.Materialize(plan, inputs, heap.ReadResident, ref snapshot, ref specialization, out _)); + Assert.Equal(2, bulkReads); + var wordReads = heap.Reads; + var reference = new ResourceSnapshot(); + var referenceSpecialization = new ResourceSpecialization(); + Assert.True(ResourceMaterializer.Materialize(plan, inputs with { ReadCleanWords = null }, + ref reference, ref referenceSpecialization)); + Assert.Equal(reference.Images.Select(image => image.ToArray()), snapshot.Images.Select(image => image.ToArray())); + Assert.Equal(16, heap.Reads - 2 * wordReads); // Two eight-dword descriptors replaced by two range reads. + Assert.True(cache.Materialize(plan, inputs, heap.ReadResident, ref snapshot, ref specialization, out _)); + Assert.Equal(1, cache.Hits); + heap.Words[HeapBase + 0x100 + 5 * 32] = 0x3000; + Assert.True(cache.Materialize(plan, inputs, heap.ReadResident, ref snapshot, ref specialization, out _)); + Assert.Equal(2, cache.Misses); + Assert.Equal(4, bulkReads); + } + + [Fact] + public void UnchangedMemoryReusesTheMaterialization() + { + var plan = Plan(); + var heap = new Heap(); + var cache = new ResourceMaterializationCache(); + Assert.True(Run(cache, plan, heap, [0x1000, 0], out var first, out var firstSpecialization)); + var readsAfterFirst = heap.Reads; + Assert.True(readsAfterFirst > 0); + + Assert.True(Run(cache, plan, heap, [0x1000, 0], out var second, out var secondSpecialization)); + Assert.Equal(readsAfterFirst, heap.Reads); + Assert.Same(first, second); + Assert.Same(firstSpecialization, secondSpecialization); + Assert.Equal((1, 1), (cache.Hits, cache.Misses)); + } + + [Fact] + public void APlanWhoseInputsNeverRepeatStopsRecordingButKeepsItsResults() + { + var plan = Plan(); + var heap = new Heap(); + var cache = new ResourceMaterializationCache(); + for (var draw = 0u; draw < 200; draw++) + { + Assert.True(Run(cache, plan, heap, [0x1000, 0], out var snapshot, out _, shaderBase: 0x10000 + draw * 0x100)); + var reference = new ResourceSnapshot(); + var referenceSpecialization = new ResourceSpecialization(); + Assert.True(ResourceMaterializer.Materialize(plan, Inputs([0x1000, 0], readCleanMemory: heap.Read, shaderBase: 0x10000 + draw * 0x100), + ref reference, ref referenceSpecialization)); + Assert.Equal(reference.Images.Select(image => image.ToArray()), snapshot.Images.Select(image => image.ToArray())); + } + + Assert.True(cache.Bypassed > 100); + Assert.Equal(0, cache.Hits); + } + + [Fact] + public void APlanThatKeepsHittingKeepsTheCache() + { + var plan = Plan(); + var heap = new Heap(); + var cache = new ResourceMaterializationCache(); + for (var draw = 0; draw < 200; draw++) + Assert.True(Run(cache, plan, heap, [0x1000, 0], out _, out _)); + + Assert.Equal(0, cache.Bypassed); + Assert.Equal(199, cache.Hits); + } + + [Fact] + public void AChangedDescriptorWordMaterializesAgain() + { + var plan = Plan(); + var heap = new Heap(); + var cache = new ResourceMaterializationCache(); + Assert.True(Run(cache, plan, heap, [0x1000, 0], out var first, out _)); + + heap.Words[HeapBase + 0x100 + 5 * 32] = 0x3000; + Assert.True(Run(cache, plan, heap, [0x1000, 0], out var second, out _)); + Assert.NotSame(first, second); + Assert.Equal((0, 2), (cache.Hits, cache.Misses)); + + // An uncached full walk gives the same result as the re-materialization. + var reference = new ResourceSnapshot(); + var referenceSpecialization = new ResourceSpecialization(); + Assert.True(ResourceMaterializer.Materialize(plan, Inputs([0x1000, 0], readCleanMemory: heap.Read), + ref reference, ref referenceSpecialization)); + Assert.Equal(reference.Images.Select(image => image.ToArray()), second.Images.Select(image => image.ToArray())); + } + + [Fact] + public void AlternatingDescriptorsReuseEveryRecentVariant() + { + var plan = Plan(); + var heap = new Heap(); + var cache = new ResourceMaterializationCache(); + var word = HeapBase + 0x100 + 5 * 32; + Assert.True(Run(cache, plan, heap, [0x1000, 0], out var first, out _)); + heap.Words[word] = 0x3000; + Assert.True(Run(cache, plan, heap, [0x1000, 0], out var second, out _)); + Assert.Equal((0, 2), (cache.Hits, cache.Misses)); + + heap.Words[word] = 0x1000; + Assert.True(Run(cache, plan, heap, [0x1000, 0], out var firstAgain, out _)); + heap.Words[word] = 0x3000; + Assert.True(Run(cache, plan, heap, [0x1000, 0], out var secondAgain, out _)); + Assert.Same(first, firstAgain); + Assert.Same(second, secondAgain); + Assert.Equal((2, 2), (cache.Hits, cache.Misses)); + } + + [Fact] + public void AChangedMaskMaterializesAgain() + { + var plan = Plan(); + var heap = new Heap(); + var cache = new ResourceMaterializationCache(); + Assert.True(Run(cache, plan, heap, [0x1000, 0], out _, out _)); + heap.Words[HeapBase + 0x80] = 1u << 1; + Assert.True(Run(cache, plan, heap, [0x1000, 0], out _, out _)); + Assert.Equal(0, cache.Hits); + } + + [Fact] + public void GpuOwnedMemoryIsNeverTrusted() + { + var plan = Plan(); + var heap = new Heap(); + var cache = new ResourceMaterializationCache(); + Assert.True(Run(cache, plan, heap, [0x1000, 0], out _, out _)); + heap.GpuOwned = true; + Assert.True(Run(cache, plan, heap, [0x1000, 0], out _, out _)); + Assert.Equal(0, cache.Hits); + } + + [Fact] + public void ADifferentDrawKeyIsADifferentEntry() + { + var plan = Plan(); + var heap = new Heap(); + var cache = new ResourceMaterializationCache(); + Assert.True(Run(cache, plan, heap, [0x1000, 0], out var first, out _)); + Assert.True(Run(cache, plan, heap, [0x1000, 0], out var second, out _, shaderBase: 0x100)); + Assert.NotSame(first, second); + Assert.True(Run(cache, plan, heap, [0x1000, 0], out var third, out _)); + Assert.Same(first, third); + Assert.Equal((1, 2), (cache.Hits, cache.Misses)); + } + + [Fact] + public void AFailedReadIsNotCached() + { + var plan = Plan(); + var heap = new Heap(); + heap.Words.Remove(HeapBase + 0x100 + 5 * 32 + 4); + var cache = new ResourceMaterializationCache(); + var ok = Run(cache, plan, heap, [0x1000, 0], out _, out _); + var reads = heap.Reads; + Assert.Equal(ok, Run(cache, plan, heap, [0x1000, 0], out _, out _)); + Assert.True(heap.Reads > reads); + Assert.Equal(0, cache.Hits); + } + + [Fact] + public void AFailedRecordingDoesNotPoisonTheNextReader() + { + var plan = Plan(); + var incomplete = new Heap(); + incomplete.Words.Remove(HeapBase + 0x100 + 5 * 32 + 4); + var cache = new ResourceMaterializationCache(); + Run(cache, plan, incomplete, [0x1000, 0], out _, out _); + + var complete = new Heap(); + Assert.True(Run(cache, plan, complete, [0x1000, 0], out var first, out _)); + Assert.True(Run(cache, plan, complete, [0x1000, 0], out var second, out _)); + Assert.Same(first, second); + Assert.Equal(1, cache.Hits); + + // Retained entries must not refer to scratch reads reused for another key. + complete.Words[HeapBase + 0x80] = 1u << 1; + Assert.True(Run(cache, plan, complete, [0x1000, 0], out _, out _, shaderBase: 0x100)); + var original = new Heap(); + Assert.True(Run(cache, plan, original, [0x1000, 0], out var restored, out _)); + Assert.Same(first, restored); + } + + // The table pointer in s[0:1]; the rest of the nine user-data registers the program declares. + private static readonly uint[] TableUserData = [0x1000, 0, 0, 0, 0, 0, 0, 0, 0]; + + // Reads the four constants of FlattenedReadReuseTests.RepeatedReadProgram from a word memory. + private static bool RunTable(ResourceMaterializationCache cache, ShaderResourcePlan plan, TestWordMemory memory, + out ResourceSnapshot snapshot) + { + snapshot = new ResourceSnapshot(); + var specialization = new ResourceSpecialization(); + return cache.Materialize(plan, Inputs(TableUserData, memory.Read, memory.Read), (address, destination, _) => + { + for (var offset = 0; offset < destination.Length; offset += 4) + { + if (!memory.Read(address + (ulong)offset, out var word)) + return false; + BitConverter.TryWriteBytes(destination[offset..], word); + } + + return true; + }, ref snapshot, ref specialization, out _); + } + + [Fact] + public void AChangedTableOnlyWordRefreshesOnlyTheTable() + { + var (plan, _, _) = Prepare(FlattenedReadReuseTests.RepeatedReadProgram(4), userDataCount: 9); + var memory = new TestWordMemory { Words = [11, 22, 33, 44] }; + var cache = new ResourceMaterializationCache(); + Assert.True(RunTable(cache, plan, memory, out var first)); + Assert.Equal([11u, 22, 33, 44], first.FlattenedResourceTable); + + memory.Words[2] = 99; + Assert.True(RunTable(cache, plan, memory, out var second)); + Assert.Equal((0, 1, 1), (cache.Hits, cache.Misses, cache.TableRefreshes)); + Assert.Equal([11u, 22, 99, 44], second.FlattenedResourceTable); + Assert.Same(first.Buffers, second.Buffers); + + // The refreshed table matches an uncached full walk, and the refreshed entry is then a hit. + var reference = new ResourceSnapshot(); + var referenceSpecialization = new ResourceSpecialization(); + Assert.True(ResourceMaterializer.Materialize(plan, Inputs(TableUserData, memory.Read, memory.Read), ref reference, ref referenceSpecialization)); + Assert.Equal(reference.FlattenedResourceTable, second.FlattenedResourceTable); + Assert.True(RunTable(cache, plan, memory, out var third)); + Assert.Same(second, third); + Assert.Equal((1, 1, 1), (cache.Hits, cache.Misses, cache.TableRefreshes)); + } + + [Fact] + public void AnUnreadableTableWordFallsBackToAFullWalk() + { + var (plan, _, _) = Prepare(FlattenedReadReuseTests.RepeatedReadProgram(4), userDataCount: 9); + var memory = new TestWordMemory { Words = [11, 22, 33, 44] }; + var cache = new ResourceMaterializationCache(); + Assert.True(RunTable(cache, plan, memory, out _)); + memory.Words[1] = 7; + memory.FailAddress = 0x1000 + 3 * 4; + RunTable(cache, plan, memory, out _); + Assert.Equal(0, cache.TableRefreshes); + } + + [Fact] + public void AFullGenerationKeepsRecentEntries() + { + var plan = Plan(); + var heap = new Heap(); + var cache = new ResourceMaterializationCache(generationCapacity: 1); + Assert.True(Run(cache, plan, heap, [0x1000, 0], out _, out _)); + Assert.True(Run(cache, plan, heap, [0x1000, 0], out _, out _, shaderBase: 0x100)); + // The first entry moved to the old generation and is still found. + Assert.True(Run(cache, plan, heap, [0x1000, 0], out _, out _)); + Assert.Equal(1, cache.Hits); + } +} diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Resources/ResourceMaterializerTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Resources/ResourceMaterializerTests.cs index 2d368ef..11a5656 100644 --- a/tests/SharpEmu.ShaderCompiler.Tests/Resources/ResourceMaterializerTests.cs +++ b/tests/SharpEmu.ShaderCompiler.Tests/Resources/ResourceMaterializerTests.cs @@ -63,7 +63,8 @@ public sealed class ResourceMaterializerTests var memory = ResourceTrackerTests.LinearMemory(); var first = ResourceTrackerTests.ImageDescriptor(); var second = first.ToArray(); - second[3] = (second[3] & 0x0FFFFFFF) | (10u << 28); + // A converted format cannot share a case with a directly sampled one. + second[1] = (second[1] & ~(0x1FFu << 20)) | (GuestImageFormat.Format11x2x10Uint << 20); ResourceTrackerTests.WriteImage(memory, 0x2000, first); ResourceTrackerTests.WriteImage(memory, 0x2020, second); memory.At(0x1000 + 36) = 1; @@ -93,7 +94,7 @@ public sealed class ResourceMaterializerTests Assert.Equal(first, failure.TableDescriptors[0]); Assert.Equal(second, failure.TableDescriptors[1]); Assert.Equal(second, failure.ImageDescriptors[1]); - Assert.NotEqual(failure.ImageSpecializations[0].Dimension, failure.ImageSpecializations[1].Dimension); + Assert.NotEqual(failure.ImageSpecializations[0].ConversionFormat, failure.ImageSpecializations[1].ConversionFormat); Assert.Equal(userData, failure.UserData); var diagnostic = Assert.IsType(failure.SelectorDiagnostic); Assert.Equal("full_domain_no_proof", diagnostic.SelectionMode); @@ -125,7 +126,8 @@ public sealed class ResourceMaterializerTests memory.At(0x1000 + 36) = 1; var first = ResourceTrackerTests.ImageDescriptor(); var second = first.ToArray(); - second[3] = (second[3] & 0x0FFFFFFF) | (10u << 28); + // A converted format cannot share a case with a directly sampled one. + second[1] = (second[1] & ~(0x1FFu << 20)) | (GuestImageFormat.Format11x2x10Uint << 20); ResourceTrackerTests.WriteImage(memory, 0x2000, first); ResourceTrackerTests.WriteImage(memory, 0x2020, second); var addresses = new List(); @@ -149,6 +151,109 @@ public sealed class ResourceMaterializerTests Assert.Equal(1u, Assert.Single(reads, read => read.ComponentIndex == 1).Value); } + // A 116-byte-stride material table with a selector offset of 4: the exhaustive probe step is + // 4 bytes, so a table of 2260+ records exceeds the probe cap. Keys 1 and 2 sit at record + // aligned offsets (4 + 116 * 3 and 4 + 116 * 2000); key 3 is a poison value at a misaligned + // offset (4 + 116 * 10 + 8) that only the exhaustive enumeration can reach. + private const uint AlignedStride = 116; + private const uint AlignedKeyOneOffset = 4 + AlignedStride * 3; + private const uint AlignedKeyTwoOffset = 4 + AlignedStride * 2000; + private const uint MisalignedKeyThreeOffset = 4 + AlignedStride * 10 + 8; + + private static (ShaderResourcePlan Plan, ResourceRuntimeInputs Inputs, uint[][] Heap) RecordAlignedTable(uint records, bool incompatibleKeyTwo = false) + { + var plan = Extract(ResourceTrackerTests.IndirectImageProgram(false, selectorStride: AlignedStride)); + var selector = plan.DescriptorSources[(int)plan.Info.Images[0].Source].IndirectImage!; + Assert.Equal(AlignedStride, selector.SelectorStride); + Assert.Equal(4u, selector.SelectorOffset); + + const ulong heapBase = 0x60000; + uint[] userData = [0x1000, AlignedStride << 16, records, 0, (uint)heapBase, 16 << 16, 16, 0, 7]; + var memory = new TestWordMemory { Base = 0x1000, Words = new uint[0x60000 / 4], RequireAlignment = true }; + var heap = new uint[4][]; + for (var key = 0; key < heap.Length; key++) + { + heap[key] = ResourceTrackerTests.ImageDescriptor(); + heap[key][0] += (uint)key; + } + + if (incompatibleKeyTwo) + { + // A converted format cannot share a case with a directly sampled one. + heap[2][1] = (heap[2][1] & ~(0x1FFu << 20)) | (GuestImageFormat.Format11x2x10Uint << 20); + } + + for (var key = 0; key < heap.Length; key++) + { + ResourceTrackerTests.WriteImage(memory, heapBase + (ulong)key * 32, heap[key]); + } + + memory.At(0x1000 + AlignedKeyOneOffset) = 1; + memory.At(0x1000 + AlignedKeyTwoOffset) = 2; + memory.At(0x1000 + MisalignedKeyThreeOffset) = 3; + return (plan, Inputs(userData, readCleanMemory: memory.Read), heap); + } + + // 2259 records keep the exhaustive probe count under the cap (65512 offsets, step 4): the + // misaligned key is found as before. 2260 records exceed it (65541) and probe only the + // record-aligned offsets instead of failing. + [Theory] + [InlineData(2259u, true)] + [InlineData(2260u, false)] + public void UnboundedSelectorProbesFallBackToRecordAlignedOffsetsOnlyPastTheCap(uint records, bool exhaustive) + { + var (plan, inputs, heap) = RecordAlignedTable(records); + var snapshot = new ResourceSnapshot(); + var specialization = new ResourceSpecialization(); + + Assert.True(ResourceMaterializer.Materialize(plan, inputs, ref snapshot, ref specialization)); + Assert.Equal(exhaustive ? 4 : 3, snapshot.Images.Length); + for (var key = 0; key < 3; key++) + { + Assert.Contains(snapshot.Images, image => image.SequenceEqual(heap[key])); + } + + Assert.Equal(exhaustive, snapshot.Images.Any(image => image.SequenceEqual(heap[3]))); + } + + [Theory] + [InlineData(2259u, false)] + [InlineData(2260u, true)] + public void RecordAlignedFallbackReportsTheProbedOffsets(uint records, bool fallback) + { + var (plan, inputs, _) = RecordAlignedTable(records, incompatibleKeyTwo: true); + var snapshot = new ResourceSnapshot(); + var specialization = new ResourceSpecialization(); + var captures = new List(); + + Assert.False(ResourceMaterializer.Materialize(plan, inputs, ref snapshot, ref specialization, + out var failureKind, captures.Add)); + Assert.Equal(ResourceMaterializationFailure.IncompatibleImageCandidates, failureKind); + var diagnostic = Assert.IsType(Assert.Single(captures).SelectorDiagnostic); + Assert.Equal(fallback ? "record_aligned_fallback" : "full_domain_no_proof", diagnostic.SelectionMode); + SelectorKeyProbe[] expected = fallback ? + [new(AlignedKeyOneOffset, 1), new(AlignedKeyTwoOffset, 2)] : + [new(AlignedKeyOneOffset, 1), new(MisalignedKeyThreeOffset, 3), new(AlignedKeyTwoOffset, 2)]; + Assert.Equal(expected, diagnostic.KeyProbes); + } + + // 70000 records leave more than the cap even among the record-aligned offsets. + [Fact] + public void UnboundedSelectorProbesStillFailWhenTheAlignedOffsetsExceedTheCap() + { + var (plan, inputs, _) = RecordAlignedTable(70000); + var snapshot = new ResourceSnapshot(); + var specialization = new ResourceSpecialization(); + var previousSnapshot = snapshot; + var previousSpecialization = specialization; + + Assert.False(ResourceMaterializer.Materialize(plan, inputs, ref snapshot, ref specialization, out var failureKind)); + Assert.Equal(ResourceMaterializationFailure.Other, failureKind); + Assert.Contains("unbounded selector probes exceed 65536", ResourceMaterializer.LastFailureDetail); + Assert.Same(previousSnapshot, snapshot); + Assert.Same(previousSpecialization, specialization); + } + private static IEnumerable ImageWords(ref uint pc, uint register, uint address, uint format) { uint[] words = [address, format << 20, 3 | (3 << 14), 0xFAC | (ImageType2D << 28), 0, 0, 0, 0]; diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Resources/ResourceTestProgram.cs b/tests/SharpEmu.ShaderCompiler.Tests/Resources/ResourceTestProgram.cs index faeeeb5..cee0488 100644 --- a/tests/SharpEmu.ShaderCompiler.Tests/Resources/ResourceTestProgram.cs +++ b/tests/SharpEmu.ShaderCompiler.Tests/Resources/ResourceTestProgram.cs @@ -173,7 +173,7 @@ internal static class ResourceTestProgram // The plan, its default specialization applied, and the layout of one program. public static (ShaderResourcePlan Plan, SpecializedResourceInfo Resources, BindingLayout Layout) Prepare( - Gen5ShaderProgram program, ShaderStage stage = ShaderStage.Compute, uint userDataBase = 0, uint userDataCount = 64, uint pushDataStartDword = 0) + Gen5ShaderProgram program, ShaderStage stage = ShaderStage.Compute, uint userDataBase = 0, uint userDataCount = 64, uint pushDataStartDword = 0, bool usesBindlessImages = false) { var plan = ShaderResourcePlan.Extract(program, stage, Hash, userDataBase, userDataCount); var resources = ResourceMaterializer.ApplyTo(plan, ResourceSpecialization.Default(plan.Info)); @@ -183,13 +183,14 @@ internal static class ResourceTestProgram BindingLayout.UsesGlobalDataShare(program), ShaderCompileRequest.RequiresFlattenedTable(plan, resources), BindingLayout.ReadsShaderBase(program), - pushDataStartDword); + pushDataStartDword, + usesBindlessImages: usesBindlessImages); return (plan, resources, layout); } - public static ShaderCompileRequest Request(Gen5ShaderProgram program, ShaderStage stage = ShaderStage.Compute, uint userDataBase = 0, uint userDataCount = 64, uint pushDataStartDword = 0) + public static ShaderCompileRequest Request(Gen5ShaderProgram program, ShaderStage stage = ShaderStage.Compute, uint userDataBase = 0, uint userDataCount = 64, uint pushDataStartDword = 0, bool usesBindlessImages = false) { - var (plan, resources, layout) = Prepare(program, stage, userDataBase, userDataCount, pushDataStartDword); + var (plan, resources, layout) = Prepare(program, stage, userDataBase, userDataCount, pushDataStartDword, usesBindlessImages); return new ShaderCompileRequest(plan, resources, layout) { PixelOutputs = stage == ShaderStage.Pixel ? [new Gen5PixelOutputBinding(0, 0, Gen5PixelOutputKind.Float)] : [], diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Resources/ResourceTrackerTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Resources/ResourceTrackerTests.cs index 1f273db..0ffb646 100644 --- a/tests/SharpEmu.ShaderCompiler.Tests/Resources/ResourceTrackerTests.cs +++ b/tests/SharpEmu.ShaderCompiler.Tests/Resources/ResourceTrackerTests.cs @@ -11,6 +11,42 @@ namespace SharpEmu.ShaderCompiler.Tests.Resources; public sealed class ResourceTrackerTests { + [Theory] + [InlineData("SCmpkGeU32", 0xFFFF, ScalarOperation.UGreaterThanEqual32, 0xFFFFu)] + [InlineData("SCmpkGeI32", 0xFFFF, ScalarOperation.SGreaterThanEqual32, 0xFFFF_FFFFu)] + public void ScalarCompareKUsesItsEncodedSourceAndImmediate( + string opcode, + ushort immediate, + ScalarOperation expectedComparison, + uint expectedImmediate) + { + var program = Program( + ScalarCompareK(0, opcode, source: 10, immediate), + Sop2(4, "SMulI32", 12, Gen5Operand.Scalar(10), Operand(384)), + EndProgram(8)); + var graph = ScalarValueGraph.Build(program, userDataBase: 10, userDataCount: 1); + + var source = Assert.Single(graph.Values, value => + value.Kind == ScalarValueKind.UserData && value.UserDataRegister == 10); + var comparison = Assert.Single(graph.Values, value => + value.Kind == ScalarValueKind.Operation && value.Operation == expectedComparison); + var product = Assert.Single(graph.Values, value => + value.Kind == ScalarValueKind.Operation && value.Operation == ScalarOperation.IMul32); + + Assert.Same(source, comparison.Operands[0]); + Assert.Equal(expectedImmediate, comparison.Operands[1].ConstantU32); + Assert.Same(source, product.Operands[0]); + Assert.Equal(384u, product.Operands[1].ConstantU32); + } + + private static Gen5ShaderInstruction ScalarCompareK(uint pc, string opcode, uint source, ushort immediate) + { + var word = ((source & 0x7Fu) << 16) | immediate; + return new Gen5ShaderInstruction(pc, Gen5ShaderEncoding.Sopk, opcode, [word], + [new Gen5Operand(Gen5OperandKind.EncodedConstant, immediate)], + [Gen5Operand.Scalar(source)], null); + } + private const uint Format32x4Float = 77; private const uint Format32x2Float = 64; private const uint ImageType2D = 9; @@ -129,7 +165,7 @@ public sealed class ResourceTrackerTests } [Fact] - public void SamplerWithDivergentBits_IsRejected() + public void SamplerWithDivergentBits_IsReadAtRuntime() { var program = Program( MoveScalarRegister(0, 16, 0), @@ -141,9 +177,14 @@ public sealed class ResourceTrackerTests Image(0x200, "ImageSample", 8, 16), EndProgram(0x208)); - var error = Assert.Throws(() => Extract(program)); - Assert.Contains("not a valid runtime value", error.Message); - Assert.Contains("pc=0x00000200", error.Message); + // The sampler has no plan-time source: the sample reads both descriptors from its + // registers through the runtime descriptor table instead of being planned. + var plan = Extract(program); + Assert.True(plan.Info.UsesRuntimeDescriptors); + var access = Assert.Single(plan.Memory.Entries, entry => entry.Pc == 0x200); + Assert.True(access.RuntimeDescriptor); + Assert.Empty(plan.Info.Images); + Assert.Empty(plan.Info.Samplers); } private static uint[] StorageDescriptorUserData(uint mipBase, uint mipLast) => @@ -215,7 +256,7 @@ public sealed class ResourceTrackerTests // The material table s[0:3], the heap s[4:7], the key selector in s8; the image words // come from the heap record the key selects. - internal static Gen5ShaderProgram IndirectImageProgram(bool malformed, int materialImmediate = 0, bool memoryBackedMaterial = false) + internal static Gen5ShaderProgram IndirectImageProgram(bool malformed, int materialImmediate = 0, bool memoryBackedMaterial = false, uint selectorStride = 224) { var instructions = new List(); uint pc = 0x1000; @@ -231,7 +272,7 @@ public sealed class ResourceTrackerTests Add(At(current => Vop1(current, "VMovB32", 1, Gen5Operand.Scalar(8)))); Add(At(current => ReadFirstLane(current, 9, 1))); - Add(At(current => Sop2(current, "SMulI32", 10, Gen5Operand.Scalar(9), Operand(224)))); + Add(At(current => Sop2(current, "SMulI32", 10, Gen5Operand.Scalar(9), Operand(selectorStride)))); Add(At(current => Sop2(current, "SAddU32", 11, Gen5Operand.Scalar(10), Operand(4)))); Add(At(current => ScalarBufferLoad(current, 0, destination: 12, immediateOffset: materialImmediate, dynamicOffsetRegister: 11))); Add(At(current => Sop2(current, "SLshlB32", 13, Gen5Operand.Scalar(12), Operand(5)))); @@ -401,12 +442,107 @@ public sealed class ResourceTrackerTests } [Fact] - public void MalformedIndirectImage_IsRejected() + public void MalformedIndirectImage_IsReadAtRuntime() { - var error = Assert.Throws(() => Extract(IndirectImageProgram(true))); + var plan = Extract(IndirectImageProgram(true)); + Assert.True(plan.Info.UsesRuntimeDescriptors); + Assert.Contains(plan.Memory.Entries, entry => entry.RuntimeDescriptor); + } + + // A storage access has no runtime view: an unresolvable descriptor still rejects the plan. + [Fact] + public void UnresolvableStorageImage_IsStillRejected() + { + var program = Program( + Vop2(0, "VLshlrevB32", 1, Operand(12), Gen5Operand.Vector(0)), + ReadFirstLane(4, 20, 1), + Sop2(8, "SOrB32", 3, Gen5Operand.Scalar(3), Gen5Operand.Scalar(20)), + Image(0x200, "ImageStore", 0), + EndProgram(0x208)); + + var error = Assert.Throws(() => Extract(program)); Assert.Contains("not a valid runtime value", error.Message); } + // A depth-compare sample has no runtime view: descriptors read from an empty scalar buffer + // are the null descriptor, and materialization rechecks that the buffer is still empty. + [Fact] + public void DynamicDescriptorsFromZeroExtentBufferAreNullOnlyWhileTheBufferIsEmpty() + { + var program = Program( + ScalarLoad(0, 0, destination: 28, count: 4), + ReadFirstLane(8, 10, 0), + ScalarBufferLoad(12, 28, destination: 16, count: 8, dynamicOffsetRegister: 10), + ScalarBufferLoad(20, 28, destination: 24, count: 4, dynamicOffsetRegister: 10), + Image(28, "ImageSampleC", 16, 24), + EndProgram(36)); + var plan = Extract(program, userDataCount: 2); + Assert.False(plan.Info.UsesRuntimeDescriptors); + Assert.NotNull(plan.DescriptorSources[(int)plan.Info.Images[0].Source].ZeroExtentBufferSource); + Assert.NotNull(plan.DescriptorSources[(int)plan.Info.Samplers[0].Source].ZeroExtentBufferSource); + + var memory = new TestWordMemory { Base = 0x1000, Words = new uint[8], RequireAlignment = true }; + memory.At(0x1000) = 0x2000; + memory.At(0x1004) = 8u << 16; + var inputs = Inputs([0x1000, 0], memory.Read, memory.Read); + var snapshot = new ResourceSnapshot(); + var specialization = new ResourceSpecialization(); + Assert.True(ResourceMaterializer.Materialize(plan, inputs, ref snapshot, ref specialization)); + Assert.All(snapshot.Images[0], word => Assert.Equal(0u, word)); + Assert.All(snapshot.Samplers[0], word => Assert.Equal(0u, word)); + + var priorSnapshot = snapshot; + memory.At(0x1008) = 1; + Assert.False(ResourceMaterializer.Materialize(plan, inputs, ref snapshot, ref specialization)); + Assert.Same(priorSnapshot, snapshot); + } + + // The runtime table reads whatever the buffer holds, so an access it can serve keeps that + // path rather than a null descriptor that a non-empty buffer would later invalidate. + [Fact] + public void RuntimeReadableDescriptorsFromZeroExtentBufferAreReadAtRuntime() + { + var program = Program( + ScalarLoad(0, 0, destination: 28, count: 4), + ReadFirstLane(8, 10, 0), + ScalarBufferLoad(12, 28, destination: 16, count: 8, dynamicOffsetRegister: 10), + ScalarBufferLoad(20, 28, destination: 24, count: 4, dynamicOffsetRegister: 10), + Image(28, "ImageSample", 16, 24), + EndProgram(36)); + var plan = Extract(program, userDataCount: 2); + + Assert.True(plan.Info.UsesRuntimeDescriptors); + Assert.DoesNotContain(plan.DescriptorSources, source => source.ZeroExtentBufferSource is not null); + } + + [Fact] + public void ZeroExtentDescriptorLoadsRemainNullAcrossInvariantLoopPhis() + { + var program = Program( + ScalarLoad(0, 0, destination: 28, count: 4), + ReadFirstLane(8, 10, 0), + ScalarBufferLoad(12, 28, destination: 16, count: 8, dynamicOffsetRegister: 10), + ScalarBufferLoad(20, 28, destination: 24, count: 4, dynamicOffsetRegister: 10), + Nop(28), + Branch(32, "SCbranchScc1", -2), + Image(36, "ImageSampleC", 16, 24), + EndProgram(44)); + var plan = Extract(program, userDataCount: 2); + Assert.False(plan.Info.UsesRuntimeDescriptors); + Assert.True(plan.Memory.TryGetIndex(36, 0, out var imageMemoryIndex)); + Assert.Equal(ScalarValueKind.Phi, plan.Accesses[imageMemoryIndex]!.Handle!.Operands[0].Kind); + Assert.NotNull(plan.DescriptorSources[(int)plan.Info.Images[0].Source].ZeroExtentBufferSource); + Assert.NotNull(plan.DescriptorSources[(int)plan.Info.Samplers[0].Source].ZeroExtentBufferSource); + + var memory = new TestWordMemory { Base = 0x1000, Words = new uint[8], RequireAlignment = true }; + memory.At(0x1000) = 0x2000; + memory.At(0x1004) = 8u << 16; + var snapshot = new ResourceSnapshot(); + var specialization = new ResourceSpecialization(); + Assert.True(ResourceMaterializer.Materialize(plan, Inputs([0x1000, 0], memory.Read, memory.Read), + ref snapshot, ref specialization)); + } + [Fact] public void MaterialKeyImmediateIsReadAfterTheWrappedSelectorOffset() { @@ -553,8 +689,10 @@ public sealed class ResourceTrackerTests BufferLoad(32, 8), EndProgram(36)); - var error = Assert.Throws(() => Extract(program)); - Assert.Contains("not a valid runtime value", error.Message); + // The control-dependent V# is read from its SGPRs when the access runs. + var plan = Extract(program); + Assert.Empty(plan.Info.Buffers); + Assert.True(plan.Info.UsesDeviceAddresses); } [Fact] diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Resources/RuntimeDescriptorEmissionTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Resources/RuntimeDescriptorEmissionTests.cs new file mode 100644 index 0000000..8944e4d --- /dev/null +++ b/tests/SharpEmu.ShaderCompiler.Tests/Resources/RuntimeDescriptorEmissionTests.cs @@ -0,0 +1,189 @@ +// Copyright (C) 2026 SharpEmu Emulator Project +// SPDX-License-Identifier: GPL-2.0-or-later + +using SharpEmu.ShaderCompiler; +using SharpEmu.ShaderCompiler.Resources; +using SharpEmu.ShaderCompiler.Vulkan; +using Xunit; +using static SharpEmu.ShaderCompiler.Tests.Resources.ResourceTestProgram; + +namespace SharpEmu.ShaderCompiler.Tests.Resources; + +public sealed class RuntimeDescriptorEmissionTests +{ + // A sample whose sampler words come from a lane-dependent value: no plan-time source. + private static Gen5ShaderProgram RuntimeSamplerProgram() => Program( + MoveScalarRegister(0, 16, 0), + MoveScalarRegister(4, 17, 1), + MoveScalarRegister(8, 18, 2), + Vop2(12, "VLshlrevB32", 1, Operand(12), Gen5Operand.Vector(0)), + ReadFirstLane(16, 20, 1), + Sop2(20, "SOrB32", 19, Gen5Operand.Scalar(3), Gen5Operand.Scalar(20)), + Image(0x200, "ImageSample", 8, 16), + EndProgram(0x208)); + + // The image descriptor itself comes from a lane-dependent value, so ImageLoad cannot + // materialize an image at plan time. Its runtime path needs typed bindless ImageFetch. + private static Gen5ShaderProgram RuntimeImageLoadProgram() => Program( + MoveScalarRegister(0, 16, 0), + MoveScalarRegister(4, 17, 1), + MoveScalarRegister(8, 18, 2), + Vop2(12, "VLshlrevB32", 1, Operand(12), Gen5Operand.Vector(0)), + ReadFirstLane(16, 20, 1), + Sop2(20, "SOrB32", 19, Gen5Operand.Scalar(3), Gen5Operand.Scalar(20)), + Image(0x200, "ImageLoad", 19, vectorAddress: 0, dmask: 1), + EndProgram(0x208)); + + // IMAGE_SAMPLE_A (MIMG opcode 0xA0) carries its adjustment in the sampler's reserved dword-3 + // bits, which select nothing in the sampler itself. + private static Gen5ShaderProgram RuntimeAdjustedSampleProgram() => Program( + MoveScalarRegister(0, 16, 0), + MoveScalarRegister(4, 17, 1), + MoveScalarRegister(8, 18, 2), + Vop2(12, "VLshlrevB32", 1, Operand(12), Gen5Operand.Vector(0)), + ReadFirstLane(16, 20, 1), + Sop2(20, "SOrB32", 19, Gen5Operand.Scalar(3), Gen5Operand.Scalar(20)), + Image(0x200, "ImageSampleA", 8, 16) with { Words = [(0xA0u & 0x7F) << 18 | (0xA0u >> 7), 0u] }, + EndProgram(0x208)); + + // An adjusted sample reads its descriptors at run time like a plain one. The key drops the + // sampler's reserved dword-3 bits, so each adjustment does not register a sampler of its own. + [Fact] + public void RuntimeDescriptorAdjustedSample_KeysTheSamplerWithoutItsReservedBits() + { + var plan = Extract(RuntimeAdjustedSampleProgram()); + Assert.True(plan.Info.UsesRuntimeDescriptors); + var resources = ResourceMaterializer.ApplyTo(plan, ResourceSpecialization.Default(plan.Info)); + var layout = BindingLayout.Allocate(resources.Info, + BindingLayout.CollectUserDataRegisters(plan.Graph.Program, 0, 64), false, + ShaderCompileRequest.RequiresFlattenedTable(plan, resources), false, + usesBindlessImages: true); + var request = new ShaderCompileRequest(plan, resources, layout) { ThreadCountX = 1, ThreadCountY = 1, ThreadCountZ = 1 }; + + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + var words = System.Runtime.InteropServices.MemoryMarshal.Cast(shader.Spirv); + Assert.Contains(~0x3FFF_F000u, words.ToArray()); + Gen5LargeDispatcherValidationTests.ValidateWithSpirvToolsWhenAvailable(shader.Spirv); + } + + [Fact] + public void RuntimeDescriptorSample_EmitsValidHeapLookups() + { + var plan = Extract(RuntimeSamplerProgram()); + Assert.True(plan.Info.UsesRuntimeDescriptors); + var resources = ResourceMaterializer.ApplyTo(plan, ResourceSpecialization.Default(plan.Info)); + var layout = BindingLayout.Allocate(resources.Info, + BindingLayout.CollectUserDataRegisters(plan.Graph.Program, 0, 64), false, + ShaderCompileRequest.RequiresFlattenedTable(plan, resources), false, + usesBindlessImages: true); + Assert.NotNull(layout.Find(DescriptorBindingKind.RuntimeDescriptorTable)); + Assert.NotNull(layout.Find(DescriptorBindingKind.RuntimeDescriptorMisses)); + + var request = new ShaderCompileRequest(plan, resources, layout) { ThreadCountX = 1, ThreadCountY = 1, ThreadCountZ = 1 }; + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + Gen5LargeDispatcherValidationTests.ValidateWithSpirvToolsWhenAvailable(shader.Spirv); + } + + [Fact] + public void RuntimeDescriptorImageLoad_EmitsTypedHeapFetches() + { + var plan = Extract(RuntimeImageLoadProgram()); + Assert.True(plan.Info.UsesRuntimeDescriptors); + Assert.Empty(plan.Info.NullDescriptorFallbacks); + var resources = ResourceMaterializer.ApplyTo(plan, ResourceSpecialization.Default(plan.Info)); + var layout = BindingLayout.Allocate(resources.Info, + BindingLayout.CollectUserDataRegisters(plan.Graph.Program, 0, 64), false, + ShaderCompileRequest.RequiresFlattenedTable(plan, resources), false, + usesBindlessImages: true); + var request = new ShaderCompileRequest(plan, resources, layout) { ThreadCountX = 1, ThreadCountY = 1, ThreadCountZ = 1 }; + + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + var module = new SpirvModuleInspector(shader.Spirv); + Assert.Contains((ushort)SpirvOp.ImageFetch, module.Opcodes); + Assert.Contains(module.Names.Values, name => name == "runtimeImagesDim2DUint"); + Assert.Contains(module.Names.Values, name => name == "runtimeImagesDim2DSint"); + Gen5LargeDispatcherValidationTests.ValidateWithSpirvToolsWhenAvailable(shader.Spirv); + } + + // Runtime descriptors index the persistent heap, so a host without it rejects the layout. + [Fact] + public void RuntimeDescriptors_NeedTheBindlessHeap() + { + var plan = Extract(RuntimeSamplerProgram()); + var resources = ResourceMaterializer.ApplyTo(plan, ResourceSpecialization.Default(plan.Info)); + var error = Assert.Throws(() => BindingLayout.Allocate(resources.Info, + BindingLayout.CollectUserDataRegisters(plan.Graph.Program, 0, 64), false, + ShaderCompileRequest.RequiresFlattenedTable(plan, resources), false)); + Assert.Contains("bindless heap", error.Message); + } + + // The host hashes keys exactly like the shader: FNV-1a over the dwords. + [Fact] + public void Hash_IsFnv1aOverTheKeyDwords() + { + Assert.Equal(RuntimeDescriptorTable.HashSeed, RuntimeDescriptorTable.Hash([])); + var expected = unchecked(((RuntimeDescriptorTable.HashSeed ^ 1u) * RuntimeDescriptorTable.HashPrime ^ 2u) * RuntimeDescriptorTable.HashPrime); + Assert.Equal(expected, RuntimeDescriptorTable.Hash([1u, 2u])); + } + + [Fact] + public void ViewClass_KeepsFloatKeysAndSeparatesIntegerViews() + { + var legacyFloat = (uint)ImageDimension.Dim2D; + var @uint = RuntimeDescriptorTable.ViewClass(ImageDimension.Dim2D, ImageNumericClass.Uint); + var sint = RuntimeDescriptorTable.ViewClass(ImageDimension.Dim2D, ImageNumericClass.Sint); + + Assert.Equal(legacyFloat, RuntimeDescriptorTable.ViewClass(ImageDimension.Dim2D)); + Assert.NotEqual(legacyFloat, @uint); + Assert.NotEqual(@uint, sint); + Assert.Equal(ImageNumericClass.Uint, RuntimeDescriptorTable.ViewNumericClass(@uint)); + Assert.Equal(ImageNumericClass.Sint, RuntimeDescriptorTable.ViewNumericClass(sint)); + } + + // The host's table answers the shader's probe for every key, through collisions. + [Fact] + public void BuiltTable_AnswersTheShadersProbeForEveryKey() + { + var images = Enumerable.Range(0, 200) + .Select(index => (Key: new uint[] { 3, (uint)index * 7919, 0xABCD, 0, 0, (uint)index, 0, 0, 0 }, Slot: (uint)index + 1)) + .ToList(); + var samplers = Enumerable.Range(0, 40) + .Select(index => (Key: new uint[] { (uint)index, 0x1234, 0, 1 }, Slot: (uint)index + 1)) + .ToList(); + var table = RuntimeDescriptorTable.Build(images, samplers); + foreach (var (key, slot) in images) + { + Assert.Equal(slot + 1, Probe(table, key, image: true)); + } + + foreach (var (key, slot) in samplers) + { + Assert.Equal(slot + 1, Probe(table, key, image: false)); + } + + Assert.Equal(0u, Probe(table, [3, 1, 2, 3, 4, 5, 6, 7, 8], image: true)); + Assert.Equal(0u, Probe(RuntimeDescriptorTable.Build([], []), [1, 2, 3, 4], image: false)); + } + + // The lookup the translator emits (LookupRuntimeDescriptor), over the table words: slot + 1, 0 when missing. + private static uint Probe(uint[] table, uint[] key, bool image) + { + uint Word(uint index) => index < table.Length ? table[index] : 0; + var capacity = Word(image ? RuntimeDescriptorTable.ImageCapacityDword : RuntimeDescriptorTable.SamplerCapacityDword); + var entries = Word(image ? RuntimeDescriptorTable.ImageEntriesDword : RuntimeDescriptorTable.SamplerEntriesDword); + var entryDwords = image ? RuntimeDescriptorTable.ImageEntryDwords : RuntimeDescriptorTable.SamplerEntryDwords; + if (capacity == 0) return 0; + var hash = RuntimeDescriptorTable.Hash(key); + for (uint probe = 0; probe < RuntimeDescriptorTable.ProbeCount; probe++) + { + var entry = entries + ((hash + probe) & (capacity - 1)) * entryDwords; + var stored = Word(entry); + if (stored != 0 && key.Select((word, index) => Word(entry + 1 + (uint)index) == word).All(match => match)) + { + return stored; + } + } + + return 0; + } +} diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Resources/ScalarValueGraphTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Resources/ScalarValueGraphTests.cs index 1f5fb43..65a2a76 100644 --- a/tests/SharpEmu.ShaderCompiler.Tests/Resources/ScalarValueGraphTests.cs +++ b/tests/SharpEmu.ShaderCompiler.Tests/Resources/ScalarValueGraphTests.cs @@ -156,8 +156,10 @@ public sealed class ScalarValueGraphTests Assert.True(RuntimeValueEvaluator.EvaluateDescriptorSource(invariant, invariant.Info.Buffers[0].Source, Inputs([]), out var result)); Assert.Equal(7u, result.Dwords[0]); - var error = Assert.Throws(() => Extract(PhiProgram(7, 9))); - Assert.Contains("not a valid runtime value", error.Message); + // A divergent V# has no single host source; the access reads it from its SGPRs. + var divergent = Extract(PhiProgram(7, 9)); + Assert.Empty(divergent.Info.Buffers); + Assert.True(divergent.Info.UsesDeviceAddresses); } [Fact] @@ -331,8 +333,10 @@ public sealed class ScalarValueGraphTests Assert.True(RuntimeValueEvaluator.FlattenResourceTable(plan, Inputs([], memory.Read), out var table)); Assert.Equal([0xA5A5A5A5u], table); + // A scalar load past the buffer's records returns zero, as on the hardware. var overflow = Extract(Read(16), userDataCount: 0); - Assert.False(RuntimeValueEvaluator.FlattenResourceTable(overflow, Inputs([], memory.Read), out _)); + Assert.True(RuntimeValueEvaluator.FlattenResourceTable(overflow, Inputs([], memory.Read), out var overflowTable)); + Assert.Equal([0u], overflowTable); } // A loop whose body runs under a divergent mask still reaches a fixpoint: the @@ -432,6 +436,24 @@ public sealed class ScalarValueGraphTests Assert.Equal([0x12345678u], table); } + [Theory] + [InlineData(0u, 0u, 2u)] + [InlineData(1u, 0u, 2u)] + [InlineData(0u, 1u, 40u)] + public void InactiveVectorWrite_PreservesSavedLane(uint low, uint high, uint lane) + { + var program = Program( + WriteLane(0, vectorRegister: 18, scalarRegister: 84, lane: lane), + MoveScalar(8, 126, low), + MoveScalar(16, 127, high), + Vop1(24, "VMovB32", 18, Gen5Operand.Scalar(0)), + ReadLane(28, scalarRegister: 84, vectorRegister: 18, lane: lane), + ScalarLoad(36, 84, destination: 4), + EndProgram(44)); + var plan = Extract(program, userDataBase: 84, userDataCount: 2); + Assert.Equal(84u, Assert.Single(plan.TableReads).Value.Operands[0].Operands[0].UserDataRegister); + } + [Fact] public void FullVectorWrite_InvalidatesSavedLanes() { diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Resources/SpirvBindingDeclarationTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Resources/SpirvBindingDeclarationTests.cs index d493b24..dd1fe41 100644 --- a/tests/SharpEmu.ShaderCompiler.Tests/Resources/SpirvBindingDeclarationTests.cs +++ b/tests/SharpEmu.ShaderCompiler.Tests/Resources/SpirvBindingDeclarationTests.cs @@ -80,6 +80,50 @@ public sealed class SpirvBindingDeclarationTests Assert.Contains((ushort)SpirvOp.AtomicOr, module.Opcodes); } + [Fact] + public void BindlessImagesUseRuntimeArraysAndASeparateResourceSet() + { + var request = Request(EveryKindProgram(), usesBindlessImages: true); + Assert.True(request.Bindings.UsesBindlessImages); + Assert.Equal(BindingLayout.ImageSlotTableDwordCount(request.Resources.Info), + (uint)request.Bindings.Descriptors + .Where(binding => ImageDescriptorBinding.ResourceClass(binding.Kind) != ImageResourceClass.None) + .Sum(binding => binding.Resources.Count)); + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + + var module = new SpirvModuleInspector(shader.Spirv); + Assert.Contains((uint)SpirvCapability.RuntimeDescriptorArray, module.Capabilities); + Assert.Contains((1u, BindingLayout.NativeBindingIndex(ShaderStage.Compute, DescriptorBindingKind.Buffers)), module.DescriptorBindings); + Assert.Contains((0u, BindingLayout.BindlessSampledImageBinding), module.DescriptorBindings); + Assert.Contains((0u, BindingLayout.BindlessStorageImageBinding), module.DescriptorBindings); + } + + [Fact] + public void BindlessPixelImagesKeepTheGlobalBindingNumbers() + { + var request = Request(EveryKindProgram(), ShaderStage.Pixel, usesBindlessImages: true); + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + + var module = new SpirvModuleInspector(shader.Spirv); + Assert.Contains((1u, BindingLayout.NativeBindingIndex(ShaderStage.Pixel, DescriptorBindingKind.Buffers)), module.DescriptorBindings); + Assert.Contains((0u, BindingLayout.BindlessSampledImageBinding), module.DescriptorBindings); + Assert.Contains((0u, BindingLayout.BindlessStorageImageBinding), module.DescriptorBindings); + Assert.DoesNotContain((0u, BindingLayout.NativeBindingIndex(ShaderStage.Pixel, DescriptorBindingKind.Samplers)), + module.DescriptorBindings); + } + + [Fact] + public void BindlessStageWithoutImagesStillUsesThePerShaderSet() + { + var request = Request(Program(BufferLoad(0, 4), EndProgram(8)), usesBindlessImages: true); + Assert.True(request.Bindings.UsesBindlessImages); + Assert.True(Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var error), error); + + var module = new SpirvModuleInspector(shader.Spirv); + Assert.Contains((1u, BindingLayout.NativeBindingIndex(ShaderStage.Compute, DescriptorBindingKind.Buffers)), module.DescriptorBindings); + Assert.DoesNotContain((0u, BindingLayout.NativeBindingIndex(ShaderStage.Compute, DescriptorBindingKind.Buffers)), module.DescriptorBindings); + } + [Fact] public void PixelStage_OffsetsEveryBindingByTheStageCount() { @@ -374,23 +418,21 @@ public sealed class SpirvBindingDeclarationTests uint firstImageType, uint secondImageType) { - AssertMixedImageCandidatesAreRejected(firstImageType, secondImageType); + AssertMismatchedCandidateBindsNull(firstImageType, secondImageType); } - private static void AssertMixedImageCandidatesAreRejected(uint firstImageType, uint secondImageType) + // A cube descriptor is not addressable through a 2D declaration (or the reverse); the + // hardware would read garbage, so that candidate binds as null instead of failing the draw. + private static void AssertMismatchedCandidateBindsNull(uint firstImageType, uint secondImageType) { var plan = Extract(IndirectImageAfterPlainImageProgram(indirectDimension: 3)); var (userData, memory) = IndirectImageAfterPlainImageInputs( 64, false, firstImageType: firstImageType, secondImageType: secondImageType); var snapshot = new ResourceSnapshot(); var specialization = new ResourceSpecialization(); - var previousSnapshot = snapshot; - var previousSpecialization = specialization; - Assert.False(ResourceMaterializer.Materialize(plan, Inputs(userData, readCleanMemory: memory.Read), - ref snapshot, ref specialization, out var failure)); - Assert.Equal(ResourceMaterializationFailure.IncompatibleImageCandidates, failure); - Assert.Same(previousSnapshot, snapshot); - Assert.Same(previousSpecialization, specialization); + Assert.True(ResourceMaterializer.Materialize(plan, Inputs(userData, readCleanMemory: memory.Read), + ref snapshot, ref specialization, out _)); + Assert.Single(new[] { snapshot.Images[1], snapshot.Images[2] }, words => words.All(word => word == 0)); } [Fact] diff --git a/tests/SharpEmu.ShaderCompiler.Tests/Resources/WaveMaskSelectorBoundsTests.cs b/tests/SharpEmu.ShaderCompiler.Tests/Resources/WaveMaskSelectorBoundsTests.cs index dfbdc5b..f472110 100644 --- a/tests/SharpEmu.ShaderCompiler.Tests/Resources/WaveMaskSelectorBoundsTests.cs +++ b/tests/SharpEmu.ShaderCompiler.Tests/Resources/WaveMaskSelectorBoundsTests.cs @@ -80,7 +80,8 @@ public sealed class WaveMaskSelectorBoundsTests memory.At(0x1000 + 5 * 224 + 4) = 1; var first = ResourceTrackerTests.ImageDescriptor(); var second = first.ToArray(); - second[3] = (second[3] & 0x0FFFFFFF) | (10u << 28); + // A converted format cannot share a case with a directly sampled one. + second[1] = (second[1] & ~(0x1FFu << 20)) | (GuestImageFormat.Format11x2x10Uint << 20); ResourceTrackerTests.WriteImage(memory, 0x2000, first); ResourceTrackerTests.WriteImage(memory, 0x2020, second); var inputs = new ResourceRuntimeInputs diff --git a/tools/SharpEmu.Tools.ShaderDump/Program.cs b/tools/SharpEmu.Tools.ShaderDump/Program.cs index 055c6ce..8b8cab6 100644 --- a/tools/SharpEmu.Tools.ShaderDump/Program.cs +++ b/tools/SharpEmu.Tools.ShaderDump/Program.cs @@ -5,6 +5,8 @@ // Unexpected decode, plan, or emission results fail the run. using System.Buffers.Binary; +using System.Text.Json; +using System.Text.RegularExpressions; using SharpEmu.HLE; using SharpEmu.ShaderCompiler; using SharpEmu.ShaderCompiler.Metal; @@ -13,6 +15,121 @@ using SharpEmu.ShaderCompiler.Vulkan; const ulong ProgramAddress = 0x100000; +// Plans a recorded program (input.ir.txt) and emits it with the default specialization: +// checks resource planning and SPIR-V emission of a shader the game rejected, offline. +// --plan-ir +if (args.Length == 4 && args[0] == "--plan-ir") +{ + var lines = File.ReadAllLines(args[1]); + var words = new List(); + uint userBase = 0, userCount = 0; + ulong hash = 0; + foreach (var line in lines) + { + var header = Regex.Match(line, @"user_data_base=(\d+) user_data_count=(\d+)"); + if (header.Success) + { + userBase = uint.Parse(header.Groups[1].Value); + userCount = uint.Parse(header.Groups[2].Value); + } + + var hashMatch = Regex.Match(line, @"hash=0x([0-9A-Fa-f]+)"); + if (hashMatch.Success && hash == 0) hash = Convert.ToUInt64(hashMatch.Groups[1].Value, 16); + var match = Regex.Match(line, @"^0x([0-9A-Fa-f]+) ([0-9A-Fa-f]{8}(?:_[0-9A-Fa-f]{8})*) "); + if (!match.Success) continue; + var pc = Convert.ToUInt32(match.Groups[1].Value, 16); + if (pc != words.Count * sizeof(uint)) + throw new ArgumentException($"Non-contiguous recorded code at 0x{pc:X}."); + words.AddRange(match.Groups[2].Value.Split('_').Select(word => Convert.ToUInt32(word, 16))); + } + + var stage = args[2] switch { "pixel" => ShaderStage.Pixel, "vertex" => ShaderStage.Vertex, _ => ShaderStage.Compute }; + var memory = new FakeMemory(); + memory.AddRegion(ProgramAddress, words.ToArray()); + if (!Gen5ShaderTranslator.TryDecodeProgram(new CpuContext(memory, Generation.Gen5), ProgramAddress, out var program, out var decodeError)) + throw new InvalidOperationException(decodeError); + var plan = ShaderResourcePlan.Extract(program!, stage, hash, userBase, userCount); + var resources = ResourceMaterializer.ApplyTo(plan, ResourceSpecialization.Default(plan.Info)); + var layout = BindingLayout.Allocate(resources.Info, + BindingLayout.CollectUserDataRegisters(program!, userBase, userCount), + BindingLayout.UsesGlobalDataShare(program!), + ShaderCompileRequest.RequiresFlattenedTable(plan, resources), + BindingLayout.ReadsShaderBase(program!), + usesBindlessImages: true, + usesRuntimeBufferStrides: true); + var request = new ShaderCompileRequest(plan, resources, layout) { ThreadCountX = 1, ThreadCountY = 1, ThreadCountZ = 1 }; + if (!Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out var compileError)) + throw new InvalidOperationException(compileError); + File.WriteAllBytes(args[3], shader.Spirv); + var runtime = plan.Memory.Entries.Count(entry => entry.RuntimeDescriptor); + Console.WriteLine($"hash=0x{hash:X16} instructions={program!.Instructions.Count} runtime_descriptor_accesses={runtime} images={resources.Info.Images.Count} spirv_bytes={shader.Spirv.Length}"); + return; +} + +// Recompile recorded compute code and its resource specialization without launching +// the game. The input IR contains original instruction words, not executable host IR. +if (args.Length == 4 && args[0] == "--replay-compute") +{ + using var capture = JsonDocument.Parse(File.ReadAllText(args[2])); + var state = capture.RootElement; + if (state.GetProperty("Stage").GetString() != "compute") + throw new ArgumentException("Only recorded compute shaders are supported."); + var words = new List(); + foreach (var line in File.ReadLines(args[1])) + { + var match = Regex.Match(line, @"^0x([0-9A-Fa-f]+) ([0-9A-Fa-f]{8}(?:_[0-9A-Fa-f]{8})*) "); + if (!match.Success) continue; + var pc = Convert.ToUInt32(match.Groups[1].Value, 16); + if (pc != words.Count * sizeof(uint)) + throw new ArgumentException($"Non-contiguous recorded code at 0x{pc:X}."); + words.AddRange(match.Groups[2].Value.Split('_').Select(word => Convert.ToUInt32(word, 16))); + } + var memory = new FakeMemory(); + memory.AddRegion(ProgramAddress, words.ToArray()); + var context = new CpuContext(memory, Generation.Gen5); + if (!Gen5ShaderTranslator.TryDecodeProgram(context, ProgramAddress, out var program, out var error)) + throw new InvalidOperationException(error); + if (program!.Instructions.Count != state.GetProperty("InstructionCount").GetInt32()) + throw new InvalidOperationException("Recorded and decoded instruction counts differ."); + var userCount = state.GetProperty("UserDataCount").GetUInt32(); + var userBase = state.GetProperty("UserDataBase").GetUInt32(); + var wave = state.GetProperty("WaveSize").GetUInt32(); + var hash = Convert.ToUInt64(state.GetProperty("ShaderHash").GetString()![2..], 16); + var plan = ShaderResourcePlan.Extract(program, ShaderStage.Compute, hash, userBase, userCount, waveSize: wave); + var specialization = state.GetProperty("Specialization").Deserialize()!; + var resources = ResourceMaterializer.ApplyTo(plan, specialization); + var layout = BindingLayout.Allocate(resources.Info, + BindingLayout.CollectUserDataRegisters(program, userBase, userCount), + BindingLayout.UsesGlobalDataShare(program), + ShaderCompileRequest.RequiresFlattenedTable(plan, resources), + BindingLayout.ReadsShaderBase(program), + state.GetProperty("PushDataStartDword").GetUInt32(), + state.GetProperty("UsesDispatchThreadLimits").GetBoolean(), + state.GetProperty("UsesBindlessImages").GetBoolean(), + state.TryGetProperty("UsesRuntimeBufferStrides", out var runtimeStrides) && runtimeStrides.GetBoolean()); + var request = new ShaderCompileRequest(plan, resources, layout) + { + WaveSize = wave, + EnableExecGuardElision = state.GetProperty("EnableExecGuardElision").GetBoolean(), + SupportsSharedInt64Atomics = state.GetProperty("SupportsSharedInt64Atomics").GetBoolean(), + SupportsExactFloat16Conversions = state.GetProperty("SupportsExactFloat16Conversions").GetBoolean(), + SupportsNonUniformImageIndexing = state.GetProperty("SupportsNonUniformImageIndexing").GetBoolean(), + ComputeSystemRegisters = state.GetProperty("ComputeSystemRegisters").Deserialize(), + ScratchDwords = state.GetProperty("ScratchDwords").GetUInt32(), + LocalSizeX = state.GetProperty("LocalSizeX").GetUInt32(), + LocalSizeY = state.GetProperty("LocalSizeY").GetUInt32(), + LocalSizeZ = state.GetProperty("LocalSizeZ").GetUInt32(), + ThreadCountX = state.GetProperty("ThreadCountX").GetUInt32(), + ThreadCountY = state.GetProperty("ThreadCountY").GetUInt32(), + ThreadCountZ = state.GetProperty("ThreadCountZ").GetUInt32(), + }; + if (!Gen5SpirvTranslator.TryCompileProgram(request, out var shader, out error)) + throw new InvalidOperationException(error); + File.WriteAllBytes(args[3], shader.Spirv); + Console.WriteLine($"hash=0x{hash:X16} instructions={program.Instructions.Count} spirv_bytes={shader.Spirv.Length}"); + return; +} + if (args.Length >= 1 && string.Equals(args[0], "--inspect", StringComparison.Ordinal)) { if (args.Length is < 2 or > 3)