FreeBSD sys_flock takes LOCK_UN over LOCK_EX over LOCK_SH, ignores other
bits and fails with EBADF when none of the three is set. The operation
reached the host unchanged, so malformed operations differed between
hosts: Linux rejects them with EINVAL and the Windows lock tracking threw.
Decode the operation before either host sees it.
flock mapped each call straight onto LockFileEx/UnlockFileEx, whose
byte-range locks stack per handle. LOCK_UN without a lock failed with
ERROR_NOT_LOCKED and aborted the process, a repeated LOCK_EX blocked
forever, converting between shared and exclusive failed, and a repeated
LOCK_SH needed as many unlocks.
Track the one lock each descriptor holds: a repeated mode and LOCK_UN
without a lock succeed, a conversion releases the old lock before taking
the new one, and close forgets the descriptor's lock.
FreeBSD's unlink(2) reports EPERM for a directory, but the host call
returns EACCES on Windows and EISDIR on Linux, so the guest saw a
different error on each host. Check for a directory, not following a
symbolic link, before unlinking.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
open, _open, stat, unlink and rmdir forwarded the result of their sceKernel
twin, so a failure returned 0x8002xxxx instead of -1 and errno was never set
to the guest error. They now return -1 with errno set to the code carried by
the sce error, and report a null pointer as EFAULT and an empty path as
ENOENT.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>