fix(libkernel): keep raises off unregistered threads and clamp the handler's MXCSR

Two defects found in review of the Linux delivery:

A raise right after scePthreadCreate reached the new thread before
RunThread set currentThread, so scePthreadSelf() in the handler adopted
a second handle, and could deadlock on glibc's thread lock when the
signal interrupted pthread_getattr_np. The creator now blocks all
signals around the std::thread construction, so the child starts with
them blocked, and RunThread restores the creator's mask once the thread
is registered, as libthr does. A raise in that window stays pending and
is delivered just before the entry runs.

A handler that set reserved MXCSR bits made rt_sigreturn fail with
SIGSEGV. The MXCSR the handler leaves is now masked with the frame's
MXCSR_MASK (0xffbf when it reads 0), as FreeBSD's fpusetregs does, on
both platforms; Windows also keeps CONTEXT.MxCsr equal to
FltSave.MxCsr.

guest_raise_exception raises right after each of 50 thread creations and
checks scePthreadSelf() in the handler, and its context round trip sets
a reserved MXCSR bit and FZ and checks that only FZ is kept.

AI-assisted: yes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LwJx7AHrSUaMGSpEGhL1Kb

core/libs/prx/libkernel/Pthread/src/Thread.cpp

@@ -23,6 +23,7 @@
 #ifndef _WIN32
 #include <cerrno>
 #include <csetjmp>
+#include <csignal>
 #include <pthread.h>
 #include <sys/mman.h>
 #include <unistd.h>
@@ -44,6 +45,9 @@ struct ThreadArgs {
     PthreadEntry entry;
     void* arg;
     PthreadPrivate* self;
+#ifndef _WIN32
+    sigset_t signalMask;
+#endif
 };
 
 static std::atomic<thread_dtors_func_t> threadDtors{nullptr};
@@ -369,7 +373,13 @@ static void RunThread(std::unique_ptr<ThreadArgs> args) {
     TimedWait::BindThreadWaitState(&self->waitCount);
     CurrentGuestState().current = self;
     RegisterStack(self);
+#ifndef _WIN32
+    const sigset_t signalMask = args->signalMask;
+#endif
     args.reset();
+#ifndef _WIN32
+    pthread_sigmask(SIG_SETMASK, &signalMask, nullptr);
+#endif
     finishThread(self, entry(arg));
     CurrentGuestState().current = nullptr;
 }
@@ -551,14 +561,22 @@ int APS5_VABI scePthreadCreate(Pthread* thread, const PthreadAttr* attr, Pthread
     if (p->stackSize < 16384 || p->stackSize > std::numeric_limits<std::size_t>::max() - reserve - HOST_STACK_MARGIN - page)
         throw std::runtime_error("scePthreadCreate: invalid stack size");
     const std::size_t nativeStack = (p->stackSize + reserve + HOST_STACK_MARGIN + page - 1) / page * page;
+    sigset_t creatorMask;
+    pthread_sigmask(SIG_SETMASK, nullptr, &creatorMask);
+    args->signalMask = creatorMask;
     auto native = std::make_unique<NativeThreadArgs>(NativeThreadArgs{std::move(args), start.get_future(), {}});
     auto initialized = native->initialized.get_future();
     pthread_attr_t nativeAttr;
     if (const int result = pthread_attr_init(&nativeAttr); result != 0)
         throw std::system_error(result, std::generic_category(), "Creating guest thread");
     int created = pthread_attr_setstacksize(&nativeAttr, nativeStack);
-    if (created == 0)
+    if (created == 0) {
+        sigset_t allSignals;
+        sigfillset(&allSignals);
+        pthread_sigmask(SIG_BLOCK, &allSignals, nullptr);
         created = pthread_create(&p->hostThread, &nativeAttr, StartNativeThread, native.get());
+        pthread_sigmask(SIG_SETMASK, &creatorMask, nullptr);
+    }
     pthread_attr_destroy(&nativeAttr);
     if (created != 0)
         throw std::system_error(created, std::generic_category(), "Creating guest thread");

core/libs/prx/libkernel/System/src/Exception.cpp

@@ -109,6 +109,28 @@ GuestExceptionHandler Handler(int signum) {
     return reinterpret_cast<GuestExceptionHandler>(handlers[signum].load(std::memory_order_acquire));
 }
 
+constexpr std::size_t MxcsrOffset = 24;
+constexpr std::size_t MxcsrMaskOffset = 28;
+constexpr std::uint32_t DefaultMxcsrMask = 0xffbf;
+
+std::uint32_t FpWord(const GuestMcontext& m, std::size_t offset) {
+    std::uint32_t value = 0;
+    std::memcpy(&value, reinterpret_cast<const unsigned char*>(m.fpstate) + offset, sizeof(value));
+    return value;
+}
+
+void SetFpWord(GuestMcontext& m, std::size_t offset, std::uint32_t value) {
+    std::memcpy(reinterpret_cast<unsigned char*>(m.fpstate) + offset, &value, sizeof(value));
+}
+
+void CallHandler(GuestExceptionHandler handler, int signum, GuestUcontext& ucontext) {
+    auto& m = ucontext.mcontext;
+    const std::uint32_t mxcsrMask = FpWord(m, MxcsrMaskOffset);
+    handler(signum, &ucontext);
+    SetFpWord(m, MxcsrOffset, FpWord(m, MxcsrOffset) & (mxcsrMask != 0 ? mxcsrMask : DefaultMxcsrMask));
+    SetFpWord(m, MxcsrMaskOffset, mxcsrMask);
+}
+
 #ifdef _WIN32
 constexpr std::size_t RedZone = 128;
 constexpr std::size_t HomeArea = 32;
@@ -154,7 +176,7 @@ void Deliver(GuestExceptionHandler handler, int signum, CONTEXT& context) {
     m.len = sizeof(GuestMcontext);
     static_assert(sizeof(context.FltSave) <= sizeof(m.fpstate));
     std::memcpy(m.fpstate, &context.FltSave, sizeof(context.FltSave));
-    handler(signum, &ucontext);
+    CallHandler(handler, signum, ucontext);
     context.Rdi = m.rdi;
     context.Rsi = m.rsi;
     context.Rdx = m.rdx;
@@ -174,6 +196,7 @@ void Deliver(GuestExceptionHandler handler, int signum, CONTEXT& context) {
     context.Rsp = m.rsp;
     context.EFlags = static_cast<DWORD>(m.rflags);
     std::memcpy(&context.FltSave, m.fpstate, sizeof(context.FltSave));
+    context.MxCsr = context.FltSave.MxCsr;
 }
 
 [[noreturn]] void RedirectedEntry(Delivery* delivery) {
@@ -344,7 +367,7 @@ void Deliver(int, siginfo_t*, void* context) {
     auto& host = static_cast<ucontext_t*>(context)->uc_mcontext;
     GuestUcontext ucontext{};
     FromHost(host, ucontext.mcontext);
-    handler(RaisedSignal, &ucontext);
+    CallHandler(handler, RaisedSignal, ucontext);
     ToHost(ucontext.mcontext, host);
 }
 

core/libs/tests/GuestRaiseException.cpp

@@ -34,6 +34,10 @@ static constexpr int Repeats = 100;
 static constexpr std::size_t R12Offset = 0xa0;
 static constexpr std::size_t R13Offset = 0xa8;
 static constexpr std::size_t Xmm8Offset = 0x140 + 160 + 8 * 16;
+static constexpr std::size_t MxcsrOffset = 0x140 + 24;
+static constexpr std::uint32_t FlushToZero = 0x8000;
+static constexpr std::uint32_t ReservedMxcsrBit = 0x10000;
+static constexpr int StartingRounds = 50;
 static constexpr std::uint64_t SpinMarker = 0x0123456789abcdefULL;
 static constexpr std::uint64_t PatchedR12 = 0x5a5a1234abcd0001ULL;
 static constexpr std::uint64_t PatchedXmm8 = 0x7e7e5678dcba0002ULL;
@@ -49,6 +53,7 @@ static std::atomic<bool> clobberVectors{false};
 static std::atomic<bool> patchContext{false};
 static std::atomic<std::uint64_t> observedR13{0};
 static std::atomic<std::uint64_t> observedXmm8{0};
+static std::atomic<std::uint32_t> observedMxcsr{0};
 
 static void APS5_VABI Handler(int signum, void* context) {
     Require(signum == SIGUSR1);
@@ -61,8 +66,13 @@ static void APS5_VABI Handler(int signum, void* context) {
         observedR13.store(value);
         std::memcpy(&value, bytes + Xmm8Offset, sizeof(value));
         observedXmm8.store(value);
+        std::uint32_t mxcsr = 0;
+        std::memcpy(&mxcsr, bytes + MxcsrOffset, sizeof(mxcsr));
+        observedMxcsr.store(mxcsr);
+        mxcsr |= FlushToZero | ReservedMxcsrBit;
         std::memcpy(bytes + R12Offset, &PatchedR12, sizeof(PatchedR12));
         std::memcpy(bytes + Xmm8Offset, &PatchedXmm8, sizeof(PatchedXmm8));
+        std::memcpy(bytes + MxcsrOffset, &mxcsr, sizeof(mxcsr));
     }
     int local = 0;
     handlerFrame.store(reinterpret_cast<std::uintptr_t>(&local));
@@ -78,9 +88,10 @@ struct SpinState {
     std::uint32_t padding = 0;
     std::uint64_t r12 = 0;
     std::uint64_t xmm8 = 0;
+    std::uint32_t mxcsr = 0;
 };
 
-static_assert(offsetof(SpinState, r12) == 8 && offsetof(SpinState, xmm8) == 16);
+static_assert(offsetof(SpinState, r12) == 8 && offsetof(SpinState, xmm8) == 16 && offsetof(SpinState, mxcsr) == 24);
 
 extern "C" void APS5_VABI RaiseTestSpinUntilR12(SpinState* state);
 asm(".text\n"
@@ -88,6 +99,8 @@ asm(".text\n"
     "RaiseTestSpinUntilR12:\n"
     "    pushq %r12\n"
     "    pushq %r13\n"
+    "    subq $8, %rsp\n"
+    "    stmxcsr (%rsp)\n"
     "    xorl %r12d, %r12d\n"
     "    movabsq $0x0123456789abcdef, %r13\n"
     "    movq %r13, %xmm8\n"
@@ -98,6 +111,9 @@ asm(".text\n"
     "    jz 1b\n"
     "    movq %r12, 8(%rdi)\n"
     "    movq %xmm8, 16(%rdi)\n"
+    "    stmxcsr 24(%rdi)\n"
+    "    ldmxcsr (%rsp)\n"
+    "    addq $8, %rsp\n"
     "    popq %r13\n"
     "    popq %r12\n"
     "    ret\n");
@@ -275,6 +291,13 @@ static void* APS5_VABI Spinning(void* arg) {
     return nullptr;
 }
 
+static std::atomic<bool> startingRelease{false};
+
+static void* APS5_VABI Starting(void*) {
+    while (!startingRelease.load()) std::this_thread::yield();
+    return nullptr;
+}
+
 static std::atomic<bool> finishedReturned{false};
 
 static void* APS5_VABI Finished(void*) {
@@ -537,6 +560,20 @@ int main() {
     patchContext.store(false);
     Require(observedR13.load() == SpinMarker && observedXmm8.load() == SpinMarker);
     Require(spin.r12 == PatchedR12 && spin.xmm8 == PatchedXmm8);
+    Require(spin.mxcsr == (observedMxcsr.load() | FlushToZero));
+
+    for (int round = 0; round < StartingRounds; ++round) {
+        const int before = calls.load();
+        startingRelease.store(false);
+        Pthread startingThread = nullptr;
+        Require(scePthreadCreate(&startingThread, nullptr, Starting, nullptr, "starting") == 0);
+        Require(sceKernelRaiseException(startingThread, SIGUSR1) == 0);
+        for (int attempt = 0; attempt < 5000 && calls.load() == before; ++attempt) std::this_thread::sleep_for(std::chrono::milliseconds(1));
+        Require(calls.load() == before + 1);
+        Require(handlerSelf.load() == startingThread);
+        startingRelease.store(true);
+        Require(scePthreadJoin(startingThread, nullptr) == 0);
+    }
 
     Pthread finishedThread = nullptr;
     Require(scePthreadCreate(&finishedThread, nullptr, Finished, nullptr, "finished") == 0);

docs/dev/TechnicalDebt.md

@@ -376,7 +376,7 @@ Throughout the project, every function at every stage either **does exactly what
 - [sceKernelSync](../../core/libs/prx/libc/src/General.cpp) (libkernel) flushes only the host files and directories recorded through `RecordWrittenPath_nid_no_patch` (the libc, libkernel, save data and app content file functions) and their changed parent directories. A host write in a library that does not record its path is not flushed by `sceKernelSync`
 - [sceKernelSync](../../core/libs/prx/libc/src/General.cpp) (libkernel) on Windows skips a recorded file it cannot open for writing (read-only, or held open by a handle that denies write sharing), since `FlushFileBuffers` needs a writable handle. On Linux, it skips a recorded file it cannot open for reading. Such a file's data may not reach the disk before a host crash; the guest reads the same data either way
 - [sceKernelInstallExceptionHandler](../../core/libs/prx/libkernel/System/src/Exception.cpp) records the handler, but a host fault still ends the process: only signals raised with `sceKernelRaiseException` are delivered. Unity's PS5Util module installs a SIGUSR1 handler that reads `uc_mcontext.mc_rsp` (offset 0xf8 of the context) and raises SIGUSR1 on other threads to stop them for garbage collection
-- [sceKernelRaiseException](../../core/libs/prx/libkernel/System/src/Exception.cpp) on Linux sends the host SIGUSR1, which the library takes over, to the target thread, so the handler runs wherever the thread was interrupted, emulator code included. On the console kernel objects are system calls and libthr defers signals while a thread holds its internal locks; here a handler that needs a host lock the interrupted code holds deadlocks, and a thread that the handler keeps waiting keeps its host locks. Host waits and sleeps resume after the handler; the host calls Linux does not restart, the poll of `sceNetEpollWait` and socket calls with a send or receive timeout, return `EINTR` to the guest
+- [sceKernelRaiseException](../../core/libs/prx/libkernel/System/src/Exception.cpp) on Linux sends the host SIGUSR1, which the library takes over, to the target thread, so the handler runs wherever the thread was interrupted, emulator code included. On the console kernel objects are system calls and libthr defers signals while a thread holds its internal locks; here a handler that needs a host lock the interrupted code holds deadlocks, and a thread that the handler keeps waiting keeps its host locks. Host waits resume after the handler. The host calls Linux does not restart end early: sleeps that do not retry return before their time, and the poll of `sceNetEpollWait` and socket calls with a send or receive timeout return `EINTR` to the guest
 - [sceKernelRaiseException](../../core/libs/prx/libkernel/System/src/Exception.cpp) builds the handler's `ucontext` with FreeBSD's amd64 layout (16-byte sigmask, `mcontext` at 0x40, `mc_rsp` at 0xf8); the console's layout is not verified beyond the fields PS5Util reads
 - [sceKernelRaiseException](../../core/libs/prx/libkernel/System/src/Exception.cpp) does not consult the target thread's signal mask
 - [pthread_sigmask](../../core/libs/prx/libkernel/System/src/Signals.cpp) (libkernel) shares the process-wide mask of `sigprocmask` and `_sigprocmask`, which keeps only the first 32 signals; FreeBSD keeps a mask per thread