GuestMemory::CollectWrites was built on the Windows arena's MEM_WRITE_WATCH
and answered 0 on Linux, so every cached copy of guest memory (textures,
resident render targets, buffers, DCC key proofs) read as changed at every
check: resident targets were stored and re-uploaded at every refresh and
unit shadows stayed off. Astro Bot's intro ran at ~3 fps.
libc gains GuestWriteWatch, the Linux counterpart of GetWriteWatch. Every
guest mapping libkernel makes (MapAligned: fixed, hinted and aligned) is
registered with one userfaultfd in asynchronous write-protect mode
(UFFD_FEATURE_WP_ASYNC: the kernel lifts a page's protection on its first
write without a handler thread, for user and kernel-mode writes alike), and
PAGEMAP_SCAN with PM_SCAN_WP_MATCHING reports the pages written since the
last scan of a range and protects them again in one call. Unmap forgets
the range. Direct memory is mapped from shared backings over its
reservation and a fixed remap replaces the mapping, so both register the
new mapping afresh. The watch is per mapping: a write through one mapping
of shared physical pages is not seen at another, so physical pages mapped
at more than one address leave the watch in every mapping of them.
A registered mapping reads as written until a collect covers it. The watch
keeps that itself instead of relying on how the kernel classifies pages
never faulted in: Linux 6.17 reports a range without page tables as not
written and leaves it unprotected, 7.2 reports it as written. The first
collect of a fresh range reports it written and write-protects it with
UFFDIO_WRITEPROTECT, which puts markers on its unpopulated pages
(UFFD_FEATURE_WP_UNPOPULATED), so from then on only real writes report.
Fallback: the userfaultfd is opened with UFFD_USER_MODE_ONLY where
vm.unprivileged_userfaultfd restricts it, and a probe at startup checks the
mechanism on scratch pages (a populated and an unpopulated page written
after protection report exactly, untouched ones do not). Without it (kernel
older than 6.7, no userfaultfd permission, headers without PAGEMAP_SCAN, or
APS5_NO_WRITE_WATCH=1) the watch reports unavailable and the tracker stays
unwatched exactly as before: CollectWrites answers 0 and callers compare
bytes. A range with a page outside the registered mappings also answers 0,
and a protection or scan that fails reports its remaining pages as written
and answers 0.
The driver's tracker keeps its contract and generations. On Linux its
block stamps live in 4 GiB leaves over the user address space, made on
first stamp, instead of the arena's flat arrays; the collect memo now works
on both platforms. GuestMemory::WriteWatched and Watched replace the arena
queries in UnitShadowEnabled and ShadowDestinationFor, so unit shadows run
on Linux over watched memory and stay off everywhere else. The Windows
arena path is unchanged apart from going through the same stamp helpers.
Host imports: a driver may lift the write protection of imported pages
itself. amdgpu validates the userptr BO behind a VK_EXT_external_memory_host
import with a writable HMM fault on every submission after the pages were
protected again, so PAGEMAP_SCAN reports them as written with no CPU store,
and a resident display image read as written at every present (found by
Pacuka on an RX 6650 XT with RADV: the presented image went black). NVIDIA
pins the pages once at the import and stays exact from then on. A probe at
device setup decides per device: it maps and watches scratch pages, collects
them clean, imports them, collects, submits a GPU copy that reads the import,
waits, and collects again. Pages written after the submission mean imports
resolve write protection: every import from then on takes its range off the
watch (GuestMemory::Unwatch) before it is made, so Watched is false for it,
CollectWrites answers 0 and callers compare bytes; GPU writes to watched
memory keep their MarkWritten stamps. The result is logged once:
"[write-watch] host imports resolve write protection: yes/no". A probe that
cannot run takes the unwatch side. APS5_WRITE_WATCH_IMPORTS=watch|unwatch
overrides the probe (default probe; anything else throws), and
APS5_NO_WRITE_WATCH=1 still turns the watch off. On the RTX 4090 the probe
reads 0 of 4 pages written after the GPU read (4 after the import itself),
so imports stay watched there.
The import itself lifts the protection once where imports stay watched:
on the RTX 4090 the pinning at vkAllocateMemory reports every imported page
as written (4 of 4 in the probe), with no CPU store. Such a report would
read as a CPU write over any GPU results pending in the range and drop
them. So every import of a watched range is made inside an import window
(GuestMemory::ImportWatched), under the tracker lock: the range is
collected right before the import, so earlier CPU writes are stamped as
usual, and again right after it, and what the window reports is stamped
"maybe written". Readers without pending results see it as a change
(UnchangedSince fails, ChangedBlocks reports it), so caches re-upload or
compare. Decisions about pending GPU results read it as untracked:
ChangedBlocks reports BlockMaybeWritten apart from BlockWritten, and the
drop rule, the write-back's keep blocks, the alias checks, the unit
shadow's stale drop and the whole-image stale checks (through the new
WrittenSince, which also answers false for a range off the watch) store
the results instead of dropping them. A real CPU store racing an import
also races the pending GPU writes to the same memory, which has no
defined order on hardware either, so the GPU results winning is
acceptable. From the second collect on the range is protected again and
watched normally. A range the watch does not cover whole, or a collect of
the window that fails, is taken off the watch instead.
An unwatched import stays off the watch for the rest of its mapping: a
remap registers the range afresh, the import kept over it takes it off
again at the next reconcile, and a re-import after an unmap follows the
decision. Unwatch invalidates the collect memo, so a collect memoized
before the import answers 0. A range that leaves the watch while a storage
image holds a generation for it no longer reads as CPU-written:
ChangedBlocks now tells whether the range is watched, and the drop rule,
the write-back's keep blocks and the alias borrow treat an unwatched range
as untracked (stored, never dropped), as a generation of 0 is.
Tests: guest_memory checks the watch through libkernel mappings (a CPU
store, a read, a kernel write through read(), another thread, a memcpy
across pages, an mprotect cycle, a partial munmap and a fixed remap) and
over a fresh 2 MiB-aligned mapping without page tables, checks direct
memory (watched, an alias takes the shared pages off the watch in both
mappings, a fixed direct remap over flexible memory is watched afresh), and
names the failing line and the reported pages. The recorder tests allocate watched
memory on both platforms, so the UnchangedSince, key proof, close race and
unit shadow tests run on Linux, and the storage refresh test gains a
watched variant: results kept on the GPU reach a CPU reader, and a plain
CPU store into one unit with newer results pending is found by the watch
alone. The recorder tests now also run the real import probe, check the
decision with the result injected (unwatch: an import leaves the watch, a
memoized collect, a partly imported range and the unimported neighbour
answer as they should, MarkWritten stamps only watched memory, a remap and
a re-import after an unmap follow the decision; watch: the import stays
watched and stamped), and check that results pending when their memory
leaves the watch are stored rather than dropped. A further test imports the
memory of a storage image with results pending in watch mode: the
results survive the import window and reach guest memory, a cache
generation taken before the import sees a change, the window reads as
maybe written and not as a CPU store, and a CPU store after it reads as
one again. Unit shadows and the
watched storage refresh report themselves as not run when imports are
compared.
As on FreeBSD, a munmap range covering more than one registered allocation (or gaps between them)
is unmapped allocation by allocation instead of throwing "unmap crosses allocation boundaries".
GTA V unmaps such ranges in its Story Mode prologue.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Starting in app0 when the folder exists made the working directory, and with it the tests, depend
on the layout next to the executable. The title opens its files by absolute /app0 paths, so the
fallback was never used: the route to story mode runs the same without it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The C11 thread layer (_Mtx_*, _Cnd_*, _Thrd_*, _Xtime_get_ticks, std::_Pad) and the libc additions
_FNan, _FDtest, _FExp, __udivti3, _Throw_Cpp_error, generic_category, localeconv, wcsncpy_s and the
aligned operator new came from the --skip-sce-module route, where our libc stood in for the title's.
With the title's sce_module loaded, its libc.prx exports every one of these NIDs and the relinker
lists the guest modules ahead of the host prx, so the imports never reach our copies. Checked both
by the NID tables (the executable imports them from libc, the guest libc.prx exports them) and by
making the four std::_Pad functions throw: the title boots, reaches the menu and loads story mode
without calling them.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
sceSaveDataSetParam dropped the params, and sceSaveDataGetParam and
sceSaveDataDirNameSearch returned zeros, so a title that checks the params of
the save it found took its profile for a new one and rewrote it with defaults
on every boot. The params now live in a <directory>.param sidecar next to the
save directory, written by SetParam (mtime refreshed) and returned by GetParam
and by the directory search, with the directory's write time when no param
was stored yet.
The mount point is /savedata<N>, as on the console: a title read its profile
through /host//_sm/0/Profile when the mount point had another shape, and that
path does not exist.
The module logs the first three calls of each entry point, like the core
module; APS5_SAVEDATA_TRACE lifts the limit.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Only the lines this branch added lose their comments; the commits before
keep them, with the reasoning behind each change, for whoever digs into
that code later.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both suites stopped at their first stale expectation, so nothing after
it ran; the BDA device tests did not link.
- the BDA device tests define the one recorder entry Resources.cpp
reaches (no recorder) and fail loudly on the others
- Buffer::Invalidate refuses memory that is not host mapped again (lost
in 8a69fef); the device tests expect it for a device-local buffer
- APS5_PIN_WAIT_MS (60000 by default, unchanged) bounds the wait before
a mutation of a leased range is refused, and the refusal names it; the
graphics tests set 200 ms, as they expect about a dozen refusals
- the expectations follow the behaviour the code chose since: a buffer's
four-DWORD V# is refused as a T#, a sampler exceeds the mock's zero
sampler slots, a V# at unmapped memory binds its committed pages only,
a rect-list fragment input without a vertex export reads as zero,
detiled rows span the padded block width, mip ranges are checked in
address order, block-compressed and 128/129/132 formats use render
target tiling, an XOR swizzle is refused only over a base that is not
64 KiB aligned, DCC and partial mip chain views decode, a cube reads as
a 2D array, the sampler fixture's max LOD fits the 4.8 field, a BDA
program with barriers is accepted, a tessellation control stage is
refused, a corrupt request fails on its signature and a newer one on
its version
- the DCC comment in the texture decoder no longer claims metadata
compression is refused
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
libkernel:
- direct memory keeps its contents per physical page across unmap and
remap; a page mapped at two live addresses at once throws
- sceKernelDirectMemoryQuery answers from tracked allocations
- SCE protections read back as set, GPU and AMPR bits included
- APR prefix path resolve, file size and stat by id, positional reads,
fsync/pread/pwrite; directories open as descriptors on Windows
- sceKernelCheckReachability, getpeername, shutdown
- Dinkumware C11 threads (_Mtx_*, _Cnd_*, _Thrd_*) in the title libc
layout; any pthread stack size, rounded at thread creation
libc:
- unaddressed guest mappings go to the system-managed area from
0x2_0000_0000, leaving the application area to fixed mappings
- the process starts in /app0
- guest sscanf on Windows; a null %s prints as (null) as on FreeBSD
- localeconv, _FNan, _FDtest, _FExp, __udivti3, wcsncpy_s, aligned
operator new, _Throw_Cpp_error
Tests: direct memory following physical pages, guest formatting.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Renames template parameters to the TKey-style the conventions doc
requires (TValue, TFunction), removes the extern "C" close comment
(not one of the two allowed structural exceptions), and replaces
the magic error/flag numbers left after the previous comment
cleanup with the existing SCE_KERNEL_ERROR_* constants and a named
RTLD_NOW constant so the code documents itself.
Comments in this codebase may only mark technical debt (per
docs/dev/CONVENTIONS.md), plus the #endif and namespace-close
structural exceptions. Removes the descriptive comments added on
this branch across PreciseWait.hpp, FormattingWide.cpp, the
direct-memory pool query/free-run plumbing, and the module loader
wiring, keeping the two allowed structural markers intact.
- Add FormattingWide.cpp implementing vswprintf_nid_postfix and
wprintf_nid_postfix, the only wide-char printf-family functions
upstream had left as unimplemented stubs (Formatting.cpp's checked
and unchecked narrow printf/scanf families are unaffected). Removes
the dead vswprintf_nid_postfix stub from Export.cpp.
- Add PreciseWait.hpp: std::condition_variable::wait_for cannot return
in under about one system tick on this MinGW toolchain regardless of
the requested budget, even with NtSetTimerResolution; PreciseSleepUs
uses a high-resolution waitable timer instead (~0.5ms vs ~15.6ms).
Header-only for now: its guest-side callers (Equeue/EventFlag/
Semaphore/AgcDriver wait loops) are out of scope for this change, so
it is not wired into them here.
fix(libc): don't walk guest frame pointers when reporting an unhandled exception
Only log __builtin_return_address(0), which doesn't depend on the guest's frames. After the fix, the log prints the exception type and the terminate handler prints what().
feat(libc): mspace memalign, malloc stats, thread-unsafe flag and nested mspaces
- Memalign uses the existing aligned allocation.
- Stats fill `SceLibcMallocManagedSize` (size checked against 0x28) from new `inUse`/`peakInUse` counters on the arena. The system size is the arena's extent.
- The thread-unsafe flag is accepted: the implementation always locks, so skipping locks is only an optimisation it doesn't need. Other flags are still rejected.
- An overlapping create is allowed only when the new range lies entirely inside a live block of the other mspace. Every other overlap is still rejected.
feat(libc): sscanf_s on linux
sscanf_s was a NotImplemented placeholder in libc/Export.cpp, and a retail title uses it to parse strings. It's now implemented on Linux. Windows still throws NotImplemented.
Brings in main up to 9958d26 (homebrew runtime, POSIX wrappers in
libkernel, sce_module guest modules and --skip-sce-module, the --to-intel
general pipeline, libSceRandom, libSceRtc, libScePosixForWebKit,
libRenoirCore, test wiring).
Resolutions where both sides implemented the same thing:
- libkernel pthread: main's structure (lazy static initialisation,
destroyed markers, MutexOperations/CondOperations, mutex protocol and
condattr clock) with this branch's TimedWait deadlines and condition
(precise timed lock/wait on Windows), wait tracing, the adaptive mutex
type 4, POSIX default ERRORCHECK mutexes, EDEADLK/EPERM results instead
of throws, and condition clocks honoured for absolute timeouts.
- libc locale: the guest Dinkumware ABI (_Locimp, facet vtables, short
ctype/case tables, locale ids, stream storage) is kept because guest code
inlines those layouts; main's C character functions are kept on top.
- libc stdio/strings/math: main's stream functions (guest FILE status
sync), strerror/perror, remove, formatting, string and math exports win
over this branch's duplicates.
- libc paths: main's working directory with this branch's path aliases
resolved on the normalised guest path.
- libkernel file: this branch's sceKernel* results as SCE errors, main's
POSIX wrappers and both fstat helpers.
- relinker: this branch's stub lowering keeps its trampolines through the
main executable path; guest modules refuse lowerings that need stubs.
- AGC driver: main's rendering wait is executed by this branch's workers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0177gthrWEBBfBu5xJDXgBfo
libkernel: a TimedWait service for condition variables, semaphores, event flags, rwlocks and
equeues with the scheduler tick; mutex/cond/sem corner cases the title hits; job worker
affinity is opt-in (APS5_JOB_AFFINITY). libc: calloc/realloc heap slot swap, a default
terminate handler with a backtrace, CPU topology helper. libSceRtc export fixes with tests.
libSceVideoOut: APS5_NO_PAD_INPUT ignores host input for unattended measurements,
APS5_FLIP_QUEUE_WAIT keeps a stalled process alive for a debugger. ApplicationHeap and
GuestRtc tests wired into the test build.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0177gthrWEBBfBu5xJDXgBfo
Exports.cpp aliased the real std::uncaught_exception(s) mangled names
directly to __cxa_uncaught_exception(s)_nid_postfix instead of the
existing _ZSt18uncaught_exceptionv_nid_postfix/_ZSt19uncaught_exceptionsv_nid_postfix
wrappers, leaving the latter as dead code and breaking the file's own
bare-name -> matching-nid_postfix alias pattern (caught by fleet QA
review, no functional impact since it's a raw symbol alias and both
sides return in the same register).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
On native Linux these PRX shared libraries and their test executables
dynamically link real libstdc++/libgcc, so a plain throw went through the
system's __cxa_throw/_Unwind_RaiseException. That still invoked this
project's own __gxx_personality_v0_nid_postfix (the CIE's personality
pointer is a raw function address, agnostic to which unwinder is walking
the stack), but handed it libgcc's incompatible private _Unwind_Context
layout, corrupting every field Personality.cpp reads and aborting with no
exception message.
Fix: OwnPersonality() now recognises the bare __gxx_personality_v0 wrapper
on Linux too (previously Windows-only), and Exports.cpp exports the real
ABI names (__cxa_throw, __cxa_begin_catch/end_catch, __cxa_rethrow,
__cxa_allocate/free_exception, __cxa_call_terminate, __cxa_get_exception_ptr,
__cxa_current_exception_type, __cxa_init_primary_exception, the std::
terminate/uncaught_exception(s) mangled names, and the core _Unwind_*
entry points) via the existing GCC_GLOBAL_ALIAS assembler-alias mechanism,
so ELF symbol interposition makes this project's own implementation win
process-wide over libgcc_s.so.1/libstdc++.so.6. __cxa_init_primary_exception
is newly implemented; GCC 13's inline std::make_exception_ptr needs it and
nothing provided it before.
Fixes 7 previously-aborting tests: guest_rendering_wait, guest_buffer_reuse,
guest_shader_link, guest_system_exit_parameters, guest_once, guest_memory,
guest_search.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
boykopovar
Michon
Dean Galvin
STL1te
STL1te
asimplestray
OverkillLabs
SamG-Coder
devilismyfriend
owner